2024-03-11T20:32:12.591 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T20:32:12.591 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591749 --owner scheduled_teuthology@teuthology --verbose --description orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests} --name teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591749/orig.config.yaml 2024-03-11T20:32:12.620 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T20:32:12.696 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591749 branch: main description: orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591749' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug ms: 1 global: mon election default strategy: 1 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c owner: scheduled_teuthology@teuthology priority: 200 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 6815 sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b sleep_before_teardown: 0 subset: 10/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c targets: smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLNbD0TUqosiT5YT4uhvbGH9R7CelOKrM6NIO6IpVk/ilbQrhC0ZVxwPLsr67GEliWP8EmtTKJuTmI5JDWbo2sE= smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvam5DfA73PCdzN8Q1OHdmrR1YiuLKklyPq47IACoU0v3cUPdkzMCYA46WGU/f/uwH+mUfOc5wtGkMY73zSqpc= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_20:08:13 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T20:32:12.697 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa; will attempt to use it 2024-03-11T20:32:12.697 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks 2024-03-11T20:32:12.698 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T20:32:12.699 INFO:teuthology.task.internal:Checking packages... 2024-03-11T20:32:12.723 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '2b0101eb74d60610d544cb8a473e23a35ac1cc0b' 2024-03-11T20:32:12.723 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T20:32:12.723 INFO:teuthology.packaging:ref: None 2024-03-11T20:32:12.723 INFO:teuthology.packaging:tag: None 2024-03-11T20:32:12.723 INFO:teuthology.packaging:branch: main 2024-03-11T20:32:12.723 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:32:12.724 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-11T20:32:12.980 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1998.g1c349451 2024-03-11T20:32:12.982 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T20:32:12.988 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T20:32:12.989 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T20:32:12.997 INFO:teuthology.task.internal:Saving configuration 2024-03-11T20:32:13.007 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T20:32:13.019 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T20:32:13.044 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591749', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:26:22.448830', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLNbD0TUqosiT5YT4uhvbGH9R7CelOKrM6NIO6IpVk/ilbQrhC0ZVxwPLsr67GEliWP8EmtTKJuTmI5JDWbo2sE='} 2024-03-11T20:32:13.066 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591749', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:26:22.446954', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvam5DfA73PCdzN8Q1OHdmrR1YiuLKklyPq47IACoU0v3cUPdkzMCYA46WGU/f/uwH+mUfOc5wtGkMY73zSqpc='} 2024-03-11T20:32:13.066 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T20:32:13.071 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-11T20:32:13.072 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-11T20:32:13.072 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T20:32:13.140 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6a0095c0d0>, signals=[15]) 2024-03-11T20:32:13.141 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T20:32:13.147 INFO:teuthology.task.internal:Opening connections... 2024-03-11T20:32:13.147 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-03-11T20:32:13.149 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:32:13.238 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2024-03-11T20:32:13.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:32:13.331 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T20:32:13.360 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-03-11T20:32:13.379 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-03-11T20:32:13.379 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-03-11T20:32:13.436 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2024-03-11T20:32:13.437 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9" 2024-03-11T20:32:13.437 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2024-03-11T20:32:13.437 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2024-03-11T20:32:13.437 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9" 2024-03-11T20:32:13.437 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9" 2024-03-11T20:32:13.437 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T20:32:13.437 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2024-03-11T20:32:13.437 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon" 2024-03-11T20:32:13.437 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T20:32:13.437 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2024-03-11T20:32:13.437 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T20:32:13.437 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T20:32:13.438 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T20:32:13.438 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-03-11T20:32:13.459 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2024-03-11T20:32:13.475 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2024-03-11T20:32:13.476 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2024-03-11T20:32:13.532 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="CentOS Stream" 2024-03-11T20:32:13.532 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="9" 2024-03-11T20:32:13.532 INFO:teuthology.orchestra.run.smithi157.stdout:ID="centos" 2024-03-11T20:32:13.532 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="rhel fedora" 2024-03-11T20:32:13.532 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="9" 2024-03-11T20:32:13.532 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el9" 2024-03-11T20:32:13.532 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T20:32:13.533 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31" 2024-03-11T20:32:13.533 INFO:teuthology.orchestra.run.smithi157.stdout:LOGO="fedora-logo-icon" 2024-03-11T20:32:13.533 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T20:32:13.533 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://centos.org/" 2024-03-11T20:32:13.533 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T20:32:13.533 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T20:32:13.533 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T20:32:13.533 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2024-03-11T20:32:13.559 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T20:32:13.568 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T20:32:13.576 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T20:32:13.577 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-03-11T20:32:13.579 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2024-03-11T20:32:13.595 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T20:32:13.600 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T20:32:13.601 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-03-11T20:32:13.639 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2024-03-11T20:32:13.743 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T20:32:13.834 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T20:32:15.416 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T20:32:15.416 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T20:32:15.416 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T20:32:15.417 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T20:32:15.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:32:15.433 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-03-11T20:32:15.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:32:15.434 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-03-11T20:32:15.490 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-361.el9.x86_64 2024-03-11T20:32:15.491 INFO:teuthology.task.kernel:Running kernel on smithi129: 5.14.0-361.el9.x86_64 2024-03-11T20:32:15.491 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum install -y kernel 2024-03-11T20:32:15.492 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-361.el9.x86_64 2024-03-11T20:32:15.492 INFO:teuthology.task.kernel:Running kernel on smithi157: 5.14.0-361.el9.x86_64 2024-03-11T20:32:15.492 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum install -y kernel 2024-03-11T20:32:17.622 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T20:32:17.840 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-11T20:32:19.248 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-11T20:32:19.323 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-11T20:32:20.446 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-11T20:32:23.330 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-11T20:33:08.743 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 409 kB/s | 19 MB 00:47 2024-03-11T20:33:13.536 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.1 MB 00:01 2024-03-11T20:33:16.186 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 10 kB/s | 15 kB 00:01 2024-03-11T20:33:20.371 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 5.1 MB/s | 21 MB 00:04 2024-03-11T20:33:25.629 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 5.2 kB/s | 1.7 kB 00:00 2024-03-11T20:33:27.836 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T20:33:27.836 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T20:33:27.926 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:33:27.935 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:33:27.935 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-03-11T20:33:27.936 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:33:27.936 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:33:27.936 INFO:teuthology.orchestra.run.smithi129.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T20:33:27.936 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-11T20:33:27.936 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T20:33:27.936 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T20:33:27.936 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T20:33:27.936 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:33:27.936 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:33:27.936 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:33:27.936 INFO:teuthology.orchestra.run.smithi129.stdout:Install 4 Packages 2024-03-11T20:33:27.936 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:33:27.936 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 100 M 2024-03-11T20:33:27.936 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 123 M 2024-03-11T20:33:27.937 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:33:32.048 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 285 kB/s | 19 MB 01:07 2024-03-11T20:33:37.981 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.1 MB 00:02 2024-03-11T20:33:38.729 INFO:teuthology.orchestra.run.smithi129.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 571 kB/s | 6.0 MB 00:10 2024-03-11T20:33:39.415 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 15 kB 00:00 2024-03-11T20:34:11.074 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 668 kB/s | 21 MB 00:31 2024-03-11T20:34:15.946 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-11T20:34:18.145 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T20:34:18.145 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T20:34:18.227 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:34:18.233 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:34:18.233 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-03-11T20:34:18.233 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:34:18.233 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:34:18.233 INFO:teuthology.orchestra.run.smithi157.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T20:34:18.233 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-11T20:34:18.234 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T20:34:18.234 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T20:34:18.234 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T20:34:18.234 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:34:18.234 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:34:18.234 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:34:18.234 INFO:teuthology.orchestra.run.smithi157.stdout:Install 4 Packages 2024-03-11T20:34:18.234 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:34:18.235 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 100 M 2024-03-11T20:34:18.235 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 123 M 2024-03-11T20:34:18.235 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:34:18.881 INFO:teuthology.orchestra.run.smithi157.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 15 MB/s | 6.0 MB 00:00 2024-03-11T20:34:20.006 INFO:teuthology.orchestra.run.smithi157.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 14 MB/s | 21 MB 00:01 2024-03-11T20:34:24.284 INFO:teuthology.orchestra.run.smithi129.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 379 kB/s | 21 MB 00:56 2024-03-11T20:34:36.388 INFO:teuthology.orchestra.run.smithi157.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 2.2 MB/s | 39 MB 00:17 2024-03-11T20:34:40.963 INFO:teuthology.orchestra.run.smithi157.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 1.5 MB/s | 33 MB 00:22 2024-03-11T20:34:40.963 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:34:40.963 INFO:teuthology.orchestra.run.smithi157.stdout:Total 4.4 MB/s | 100 MB 00:22 2024-03-11T20:34:41.845 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:34:42.227 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:34:42.227 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:34:43.577 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:34:43.578 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:34:46.554 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:34:47.381 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T20:34:47.530 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T20:34:50.376 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T20:34:50.567 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T20:34:55.325 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T20:34:55.461 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T20:35:00.269 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T20:35:08.030 INFO:teuthology.orchestra.run.smithi129.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 383 kB/s | 33 MB 01:29 2024-03-11T20:35:10.138 INFO:teuthology.orchestra.run.smithi129.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 394 kB/s | 39 MB 01:42 2024-03-11T20:35:10.138 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:35:10.139 INFO:teuthology.orchestra.run.smithi129.stdout:Total 997 kB/s | 100 MB 01:42 2024-03-11T20:35:11.019 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:35:11.407 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:35:11.407 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:35:12.746 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:35:12.747 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:35:15.673 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:35:16.316 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T20:35:16.447 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T20:35:19.234 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T20:35:19.414 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T20:35:24.174 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T20:35:24.310 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T20:35:29.053 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T20:35:29.643 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T20:35:29.643 INFO:teuthology.orchestra.run.smithi157.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T20:35:29.643 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:35:29.730 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T20:35:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T20:35:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T20:35:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T20:35:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T20:35:30.629 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T20:35:30.630 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:35:30.630 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:35:30.630 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T20:35:30.630 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T20:35:30.630 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T20:35:30.630 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T20:35:30.630 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:35:30.630 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:35:31.149 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T20:35:31.291 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T20:35:31.291 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T20:35:31.291 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T20:35:31.291 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T20:35:31.291 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T20:35:31.292 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-11T20:35:31.292 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T20:35:31.292 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T20:35:31.413 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T20:35:31.414 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T20:35:31.414 DEBUG:teuthology.orchestra.run.smithi157:> sudo rpm -qi grub2-tools 2024-03-11T20:35:31.457 INFO:teuthology.orchestra.run.smithi157.stdout:Name : grub2-tools 2024-03-11T20:35:31.458 INFO:teuthology.orchestra.run.smithi157.stdout:Epoch : 1 2024-03-11T20:35:31.458 INFO:teuthology.orchestra.run.smithi157.stdout:Version : 2.06 2024-03-11T20:35:31.458 INFO:teuthology.orchestra.run.smithi157.stdout:Release : 68.el9 2024-03-11T20:35:31.458 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: x86_64 2024-03-11T20:35:31.458 INFO:teuthology.orchestra.run.smithi157.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T20:35:31.458 INFO:teuthology.orchestra.run.smithi157.stdout:Group : Unspecified 2024-03-11T20:35:31.458 INFO:teuthology.orchestra.run.smithi157.stdout:Size : 8264254 2024-03-11T20:35:31.458 INFO:teuthology.orchestra.run.smithi157.stdout:License : GPLv3+ 2024-03-11T20:35:31.458 INFO:teuthology.orchestra.run.smithi157.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T20:35:31.458 INFO:teuthology.orchestra.run.smithi157.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T20:35:31.458 INFO:teuthology.orchestra.run.smithi157.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T20:35:31.458 INFO:teuthology.orchestra.run.smithi157.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T20:35:31.459 INFO:teuthology.orchestra.run.smithi157.stdout:Packager : builder@centos.org 2024-03-11T20:35:31.459 INFO:teuthology.orchestra.run.smithi157.stdout:Vendor : CentOS 2024-03-11T20:35:31.459 INFO:teuthology.orchestra.run.smithi157.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T20:35:31.459 INFO:teuthology.orchestra.run.smithi157.stdout:Summary : Support tools for GRUB. 2024-03-11T20:35:31.459 INFO:teuthology.orchestra.run.smithi157.stdout:Description : 2024-03-11T20:35:31.459 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:35:31.459 INFO:teuthology.orchestra.run.smithi157.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T20:35:31.459 INFO:teuthology.orchestra.run.smithi157.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T20:35:31.459 INFO:teuthology.orchestra.run.smithi157.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T20:35:31.459 INFO:teuthology.orchestra.run.smithi157.stdout:hardware devices. 2024-03-11T20:35:31.459 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:35:31.459 INFO:teuthology.orchestra.run.smithi157.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T20:35:31.460 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T20:35:31.460 INFO:teuthology.task.kernel:Updating grub on smithi157 to boot 5.14.0-427.el9.x86_64 2024-03-11T20:35:31.460 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T20:35:32.094 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2024-03-11T20:35:33.408 INFO:teuthology.orchestra.run.smithi157.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T20:35:33.442 INFO:teuthology.orchestra.run.smithi157.stderr:done 2024-03-11T20:35:33.444 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-03-11T20:35:33.459 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.oxQ2TtvVp5 2024-03-11T20:35:33.459 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.oxQ2TtvVp5 2024-03-11T20:35:33.529 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.oxQ2TtvVp5 2024-03-11T20:35:33.662 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.oxQ2TtvVp5 is 6KB 2024-03-11T20:35:33.682 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.oxQ2TtvVp5 2024-03-11T20:35:33.696 DEBUG:teuthology.orchestra.run.smithi157:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T20:35:33.761 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T20:35:33.761 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T20:35:33.761 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T20:35:33.761 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T20:35:33.763 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T20:35:33.915 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2024-03-11T20:35:58.209 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T20:35:58.209 INFO:teuthology.orchestra.run.smithi129.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T20:35:58.209 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:35:58.294 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T20:35:58.712 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T20:35:58.712 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T20:35:58.712 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T20:35:58.712 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T20:35:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T20:35:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:35:59.142 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:35:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T20:35:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T20:35:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T20:35:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T20:35:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:35:59.142 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:35:59.655 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T20:35:59.887 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T20:35:59.887 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T20:35:59.888 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T20:35:59.888 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T20:35:59.888 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T20:35:59.888 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-11T20:35:59.888 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T20:35:59.888 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T20:36:00.009 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T20:36:00.009 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T20:36:00.010 DEBUG:teuthology.orchestra.run.smithi129:> sudo rpm -qi grub2-tools 2024-03-11T20:36:00.047 INFO:teuthology.orchestra.run.smithi129.stdout:Name : grub2-tools 2024-03-11T20:36:00.047 INFO:teuthology.orchestra.run.smithi129.stdout:Epoch : 1 2024-03-11T20:36:00.047 INFO:teuthology.orchestra.run.smithi129.stdout:Version : 2.06 2024-03-11T20:36:00.047 INFO:teuthology.orchestra.run.smithi129.stdout:Release : 68.el9 2024-03-11T20:36:00.047 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: x86_64 2024-03-11T20:36:00.047 INFO:teuthology.orchestra.run.smithi129.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T20:36:00.047 INFO:teuthology.orchestra.run.smithi129.stdout:Group : Unspecified 2024-03-11T20:36:00.048 INFO:teuthology.orchestra.run.smithi129.stdout:Size : 8264254 2024-03-11T20:36:00.048 INFO:teuthology.orchestra.run.smithi129.stdout:License : GPLv3+ 2024-03-11T20:36:00.048 INFO:teuthology.orchestra.run.smithi129.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T20:36:00.048 INFO:teuthology.orchestra.run.smithi129.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T20:36:00.048 INFO:teuthology.orchestra.run.smithi129.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T20:36:00.048 INFO:teuthology.orchestra.run.smithi129.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T20:36:00.048 INFO:teuthology.orchestra.run.smithi129.stdout:Packager : builder@centos.org 2024-03-11T20:36:00.048 INFO:teuthology.orchestra.run.smithi129.stdout:Vendor : CentOS 2024-03-11T20:36:00.048 INFO:teuthology.orchestra.run.smithi129.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T20:36:00.048 INFO:teuthology.orchestra.run.smithi129.stdout:Summary : Support tools for GRUB. 2024-03-11T20:36:00.048 INFO:teuthology.orchestra.run.smithi129.stdout:Description : 2024-03-11T20:36:00.048 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:36:00.049 INFO:teuthology.orchestra.run.smithi129.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T20:36:00.049 INFO:teuthology.orchestra.run.smithi129.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T20:36:00.049 INFO:teuthology.orchestra.run.smithi129.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T20:36:00.049 INFO:teuthology.orchestra.run.smithi129.stdout:hardware devices. 2024-03-11T20:36:00.049 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:36:00.049 INFO:teuthology.orchestra.run.smithi129.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T20:36:00.050 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T20:36:00.050 INFO:teuthology.task.kernel:Updating grub on smithi129 to boot 5.14.0-427.el9.x86_64 2024-03-11T20:36:00.050 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T20:36:00.681 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2024-03-11T20:36:02.126 INFO:teuthology.orchestra.run.smithi129.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T20:36:02.147 INFO:teuthology.orchestra.run.smithi129.stderr:done 2024-03-11T20:36:02.149 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-03-11T20:36:02.165 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.lvCTvciGrn 2024-03-11T20:36:02.165 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lvCTvciGrn 2024-03-11T20:36:02.233 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.lvCTvciGrn 2024-03-11T20:36:02.375 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.lvCTvciGrn is 6KB 2024-03-11T20:36:02.423 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.lvCTvciGrn 2024-03-11T20:36:02.438 DEBUG:teuthology.orchestra.run.smithi129:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T20:36:02.502 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T20:36:02.502 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T20:36:02.503 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T20:36:02.503 INFO:teuthology.orchestra.run.smithi129.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T20:36:02.504 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T20:36:02.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2024-03-11T20:36:03.920 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-11T20:36:03.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-11T20:36:03.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:36:32.686 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-11T20:36:32.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-11T20:36:32.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:36:38.559 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-11T20:36:47.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-11T20:36:47.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:36:50.626 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-11T20:37:02.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-11T20:37:02.634 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:37:21.055 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-11T20:37:32.698 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T20:37:36.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-11T20:37:36.059 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:37:36.363 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-03-11T20:37:36.781 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-11T20:37:36.781 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-11T20:37:36.782 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-03-11T20:37:36.795 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-427.el9.x86_64 2024-03-11T20:37:36.796 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T20:37:36.796 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T20:37:36.796 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T20:37:37.796 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-11T20:37:37.797 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T20:37:37.858 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2024-03-11T20:37:37.910 DEBUG:teuthology.parallel:result is None 2024-03-11T20:37:41.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-11T20:37:41.701 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:37:42.039 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-03-11T20:37:42.468 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-11T20:37:42.469 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-11T20:37:42.469 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-03-11T20:37:42.482 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-427.el9.x86_64 2024-03-11T20:37:42.483 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T20:37:42.483 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T20:37:42.483 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T20:37:43.484 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-11T20:37:43.484 DEBUG:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T20:37:43.545 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2024-03-11T20:37:43.588 DEBUG:teuthology.parallel:result is None 2024-03-11T20:37:43.588 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T20:37:43.596 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T20:37:43.596 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T20:37:43.598 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T20:37:43.613 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T20:37:43.619 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T20:37:43.627 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T20:37:43.628 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T20:37:43.659 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T20:37:43.688 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T20:37:43.694 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T20:37:43.695 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T20:37:43.716 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T20:37:43.751 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T20:37:43.758 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T20:37:43.764 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T20:37:43.772 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T20:37:43.774 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T20:37:43.781 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T20:37:43.782 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T20:37:43.783 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T20:37:43.833 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T20:37:43.843 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T20:37:43.844 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T20:37:43.849 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T20:37:43.889 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T20:37:43.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T20:37:44.014 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T20:37:44.055 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T20:37:44.098 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:37:44.098 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T20:37:44.176 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T20:37:44.219 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T20:37:44.273 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T20:37:44.356 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T20:37:44.396 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-11T20:37:44.396 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T20:37:44.462 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-03-11T20:37:44.465 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2024-03-11T20:37:44.506 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T20:37:44.552 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T20:37:45.064 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T20:37:45.072 INFO:teuthology.task.internal:Starting timer... 2024-03-11T20:37:45.072 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T20:37:45.082 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T20:37:45.090 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-11T20:37:45.091 DEBUG:teuthology.orchestra.run.smithi129:> sudo service auditd rotate 2024-03-11T20:37:45.135 INFO:teuthology.orchestra.run.smithi129.stdout:Rotating logs: 2024-03-11T20:37:45.137 DEBUG:teuthology.orchestra.run.smithi157:> sudo service auditd rotate 2024-03-11T20:37:45.181 INFO:teuthology.orchestra.run.smithi157.stdout:Rotating logs: 2024-03-11T20:37:45.183 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T20:37:45.184 DEBUG:teuthology.orchestra.run.smithi129:> /usr/sbin/getenforce 2024-03-11T20:37:45.205 INFO:teuthology.orchestra.run.smithi129.stdout:Permissive 2024-03-11T20:37:45.205 DEBUG:teuthology.orchestra.run.smithi157:> /usr/sbin/getenforce 2024-03-11T20:37:45.243 INFO:teuthology.orchestra.run.smithi157.stdout:Permissive 2024-03-11T20:37:45.243 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi129.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi157.front.sepia.ceph.com': 'permissive'} 2024-03-11T20:37:45.243 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-11T20:37:45.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:37:45.273 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-11T20:37:45.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:37:45.311 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T20:37:45.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo /usr/sbin/setenforce permissive 2024-03-11T20:37:45.349 DEBUG:teuthology.orchestra.run.smithi157:> sudo /usr/sbin/setenforce permissive 2024-03-11T20:37:45.387 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T20:37:45.396 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T20:37:45.413 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T20:37:45.415 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi129.front.sepia.ceph.com,smithi157.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T20:41:57.656 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi129.front.sepia.ceph.com'), Remote(name='ubuntu@smithi157.front.sepia.ceph.com')] 2024-03-11T20:41:57.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-11T20:41:57.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:41:57.748 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-03-11T20:41:57.820 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-03-11T20:41:57.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-11T20:41:57.821 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T20:41:57.912 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-03-11T20:41:57.947 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-11T20:41:57.948 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T20:41:57.959 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T20:41:57.959 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T20:41:57.959 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T20:41:57.961 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T20:41:57.961 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T20:41:57.990 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T20:41:58.006 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T20:41:58.021 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T20:41:58.035 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: ntpd: command not found 2024-03-11T20:41:58.038 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T20:41:58.048 INFO:teuthology.orchestra.run.smithi129.stdout:506 Cannot talk to daemon 2024-03-11T20:41:58.064 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T20:41:58.067 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: ntpd: command not found 2024-03-11T20:41:58.080 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T20:41:58.081 INFO:teuthology.orchestra.run.smithi157.stdout:506 Cannot talk to daemon 2024-03-11T20:41:58.098 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T20:41:58.114 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T20:41:58.129 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-03-11T20:41:58.132 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T20:41:58.132 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-03-11T20:41:58.132 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T20:41:58.132 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T20:41:58.132 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T20:41:58.132 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T20:41:58.169 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-03-11T20:41:58.173 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T20:41:58.173 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-03-11T20:41:58.173 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T20:41:58.173 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T20:41:58.173 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T20:41:58.173 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T20:41:58.174 INFO:teuthology.run_tasks:Running task pexec... 2024-03-11T20:41:58.182 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-11T20:41:58.183 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi129.front.sepia.ceph.com 2024-03-11T20:41:58.183 DEBUG:teuthology.orchestra.run.smithi129:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T20:41:58.183 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi157.front.sepia.ceph.com 2024-03-11T20:41:58.184 DEBUG:teuthology.orchestra.run.smithi157:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T20:41:58.762 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:48 ago on Mon 11 Mar 2024 08:41:10 PM UTC. 2024-03-11T20:41:58.791 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:48 ago on Mon 11 Mar 2024 08:41:10 PM UTC. 2024-03-11T20:41:58.908 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:41:58.929 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:41:58.929 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-03-11T20:41:58.929 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:41:58.929 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:41:58.929 INFO:teuthology.orchestra.run.smithi129.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T20:41:58.929 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:41:58.929 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:41:58.929 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:41:58.929 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-03-11T20:41:58.929 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:41:58.934 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 3.1 M 2024-03-11T20:41:58.934 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 9.8 M 2024-03-11T20:41:58.934 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:41:58.937 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:41:58.951 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:41:58.952 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-03-11T20:41:58.952 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:41:58.952 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:41:58.952 INFO:teuthology.orchestra.run.smithi157.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T20:41:58.952 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:41:58.952 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:41:58.952 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:41:58.952 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-11T20:41:58.952 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:41:58.959 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 3.1 M 2024-03-11T20:41:58.959 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 9.8 M 2024-03-11T20:41:58.960 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:41:59.692 INFO:teuthology.orchestra.run.smithi157.stdout:runc-1.1.12-2.el9.x86_64.rpm 5.3 MB/s | 3.1 MB 00:00 2024-03-11T20:41:59.692 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:41:59.693 INFO:teuthology.orchestra.run.smithi157.stdout:Total 4.3 MB/s | 3.1 MB 00:00 2024-03-11T20:41:59.728 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:41:59.734 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:41:59.734 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:41:59.813 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:41:59.813 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:41:59.934 INFO:teuthology.orchestra.run.smithi129.stdout:runc-1.1.12-2.el9.x86_64.rpm 3.4 MB/s | 3.1 MB 00:00 2024-03-11T20:41:59.934 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:41:59.934 INFO:teuthology.orchestra.run.smithi129.stdout:Total 3.1 MB/s | 3.1 MB 00:00 2024-03-11T20:41:59.969 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:41:59.974 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:41:59.975 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:42:00.034 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:42:00.054 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:42:00.055 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:42:00.120 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T20:42:00.284 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:42:00.378 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T20:42:00.689 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T20:42:00.985 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T20:42:00.985 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:42:00.985 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:42:00.985 INFO:teuthology.orchestra.run.smithi157.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T20:42:00.985 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:42:00.985 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:42:01.020 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T20:42:01.230 DEBUG:teuthology.parallel:result is None 2024-03-11T20:42:01.359 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T20:42:01.359 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:42:01.359 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:42:01.359 INFO:teuthology.orchestra.run.smithi129.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T20:42:01.360 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:42:01.360 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:42:01.641 DEBUG:teuthology.parallel:result is None 2024-03-11T20:42:01.642 INFO:teuthology.run_tasks:Running task install... 2024-03-11T20:42:01.650 DEBUG:teuthology.task.install:project ceph 2024-03-11T20:42:01.650 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'}} 2024-03-11T20:42:01.650 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'} 2024-03-11T20:42:01.650 INFO:teuthology.task.install:Using flavor: default 2024-03-11T20:42:01.656 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-11T20:42:01.656 INFO:teuthology.task.install:extra packages: [] 2024-03-11T20:42:01.656 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'tag': None, 'wait_for_package': False} 2024-03-11T20:42:01.657 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:42:01.659 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'tag': None, 'wait_for_package': False} 2024-03-11T20:42:01.659 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:42:01.809 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/ 2024-03-11T20:42:01.810 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1979.g2b0101eb 2024-03-11T20:42:01.878 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/ 2024-03-11T20:42:01.878 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1979.g2b0101eb 2024-03-11T20:42:01.982 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T20:42:01.983 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:42:01.983 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T20:42:02.010 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-11T20:42:02.010 DEBUG:teuthology.orchestra.run.smithi129:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T20:42:02.055 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T20:42:02.055 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-11T20:42:02.055 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T20:42:02.078 DEBUG:teuthology.orchestra.run.smithi129:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T20:42:02.087 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-11T20:42:02.087 DEBUG:teuthology.orchestra.run.smithi157:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T20:42:02.158 DEBUG:teuthology.orchestra.run.smithi129:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T20:42:02.159 DEBUG:teuthology.orchestra.run.smithi157:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T20:42:02.224 INFO:teuthology.orchestra.run.smithi129.stdout:check_obsoletes = 1 2024-03-11T20:42:02.225 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum clean all 2024-03-11T20:42:02.242 DEBUG:teuthology.orchestra.run.smithi157:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T20:42:02.310 INFO:teuthology.orchestra.run.smithi157.stdout:check_obsoletes = 1 2024-03-11T20:42:02.312 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum clean all 2024-03-11T20:42:02.466 INFO:teuthology.orchestra.run.smithi129.stdout:74 files removed 2024-03-11T20:42:02.496 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-radosgw 2024-03-11T20:42:02.572 INFO:teuthology.orchestra.run.smithi157.stdout:74 files removed 2024-03-11T20:42:02.599 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-radosgw 2024-03-11T20:42:03.291 INFO:teuthology.orchestra.run.smithi129.stdout:ceph packages for x86_64 178 kB/s | 82 kB 00:00 2024-03-11T20:42:03.399 INFO:teuthology.orchestra.run.smithi157.stdout:ceph packages for x86_64 173 kB/s | 82 kB 00:00 2024-03-11T20:42:03.616 INFO:teuthology.orchestra.run.smithi129.stdout:ceph noarch packages 36 kB/s | 10 kB 00:00 2024-03-11T20:42:03.824 INFO:teuthology.orchestra.run.smithi157.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-11T20:42:03.975 INFO:teuthology.orchestra.run.smithi129.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-11T20:42:04.241 INFO:teuthology.orchestra.run.smithi157.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-11T20:42:04.494 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T20:42:04.744 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T20:42:05.945 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-11T20:42:06.227 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-11T20:42:06.802 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-11T20:42:10.445 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 7.4 MB/s | 19 MB 00:02 2024-03-11T20:42:14.640 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.1 MB 00:00 2024-03-11T20:42:16.305 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-03-11T20:42:16.550 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 823 kB/s | 8.0 MB 00:09 2024-03-11T20:42:18.310 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-11T20:42:18.326 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-03-11T20:42:23.223 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-03-11T20:42:23.319 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.1 MB 00:01 2024-03-11T20:42:24.878 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 15 kB 00:00 2024-03-11T20:42:25.811 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:42:25.813 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:42:25.813 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-11T20:42:25.813 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:42:25.813 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:42:25.813 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 9.8 M 2024-03-11T20:42:25.814 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrading: 2024-03-11T20:42:25.814 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.4 M 2024-03-11T20:42:25.814 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.1 M 2024-03-11T20:42:25.814 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-11T20:42:25.814 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options 2024-03-11T20:42:25.814 INFO:teuthology.orchestra.run.smithi129.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T20:42:25.814 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 5.3 M 2024-03-11T20:42:25.814 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 20 M 2024-03-11T20:42:25.814 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 25 k 2024-03-11T20:42:25.814 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T20:42:25.814 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T20:42:25.815 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T20:42:25.815 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 745 k 2024-03-11T20:42:25.815 INFO:teuthology.orchestra.run.smithi129.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T20:42:25.815 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T20:42:25.815 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T20:42:25.815 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 501 k 2024-03-11T20:42:25.815 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T20:42:25.815 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 5.2 M 2024-03-11T20:42:25.815 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T20:42:25.815 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T20:42:25.815 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T20:42:25.815 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse 2024-03-11T20:42:25.815 INFO:teuthology.orchestra.run.smithi129.stdout: x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 45 k 2024-03-11T20:42:25.815 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 131 k 2024-03-11T20:42:25.816 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 162 k 2024-03-11T20:42:25.816 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T20:42:25.816 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 319 k 2024-03-11T20:42:25.816 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 298 k 2024-03-11T20:42:25.816 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 99 k 2024-03-11T20:42:25.816 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T20:42:25.816 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T20:42:25.816 INFO:teuthology.orchestra.run.smithi129.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T20:42:25.816 INFO:teuthology.orchestra.run.smithi129.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T20:42:25.816 INFO:teuthology.orchestra.run.smithi129.stdout:Installing weak dependencies: 2024-03-11T20:42:25.816 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T20:42:25.816 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:42:25.816 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:42:25.816 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:42:25.817 INFO:teuthology.orchestra.run.smithi129.stdout:Install 30 Packages 2024-03-11T20:42:25.817 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrade 2 Packages 2024-03-11T20:42:25.817 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:42:25.817 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 58 M 2024-03-11T20:42:25.817 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:42:25.913 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-11T20:42:27.557 INFO:teuthology.orchestra.run.smithi129.stdout:(1/32): ceph-radosgw-19.0.0-1979.g2b0101eb.el9. 7.5 MB/s | 9.8 MB 00:01 2024-03-11T20:42:27.599 INFO:teuthology.orchestra.run.smithi129.stdout:(2/32): ceph-selinux-19.0.0-1979.g2b0101eb.el9. 607 kB/s | 25 kB 00:00 2024-03-11T20:42:27.724 INFO:teuthology.orchestra.run.smithi129.stdout:(3/32): libcephfs2-19.0.0-1979.g2b0101eb.el9.x8 5.8 MB/s | 745 kB 00:00 2024-03-11T20:42:27.808 INFO:teuthology.orchestra.run.smithi129.stdout:(4/32): libradosstriper1-19.0.0-1979.g2b0101eb. 5.9 MB/s | 501 kB 00:00 2024-03-11T20:42:27.908 INFO:teuthology.orchestra.run.smithi129.stdout:(5/32): ceph-base-19.0.0-1979.g2b0101eb.el9.x86 3.2 MB/s | 5.3 MB 00:01 2024-03-11T20:42:28.208 INFO:teuthology.orchestra.run.smithi129.stdout:(6/32): ceph-common-19.0.0-1979.g2b0101eb.el9.x 10 MB/s | 20 MB 00:01 2024-03-11T20:42:28.234 INFO:teuthology.orchestra.run.smithi129.stdout:(7/32): python3-ceph-argparse-19.0.0-1979.g2b01 139 kB/s | 45 kB 00:00 2024-03-11T20:42:28.259 INFO:teuthology.orchestra.run.smithi129.stdout:(8/32): python3-ceph-common-19.0.0-1979.g2b0101 2.6 MB/s | 131 kB 00:00 2024-03-11T20:42:28.351 INFO:teuthology.orchestra.run.smithi129.stdout:(9/32): librgw2-19.0.0-1979.g2b0101eb.el9.x86_6 9.5 MB/s | 5.2 MB 00:00 2024-03-11T20:42:28.368 INFO:teuthology.orchestra.run.smithi129.stdout:(10/32): python3-cephfs-19.0.0-1979.g2b0101eb.e 1.2 MB/s | 162 kB 00:00 2024-03-11T20:42:28.393 INFO:teuthology.orchestra.run.smithi129.stdout:(11/32): python3-rados-19.0.0-1979.g2b0101eb.el 2.3 MB/s | 319 kB 00:00 2024-03-11T20:42:28.419 INFO:teuthology.orchestra.run.smithi129.stdout:(12/32): python3-rbd-19.0.0-1979.g2b0101eb.el9. 4.3 MB/s | 298 kB 00:00 2024-03-11T20:42:28.468 INFO:teuthology.orchestra.run.smithi129.stdout:(13/32): python3-rgw-19.0.0-1979.g2b0101eb.el9. 989 kB/s | 99 kB 00:00 2024-03-11T20:42:28.552 INFO:teuthology.orchestra.run.smithi129.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 667 kB/s | 106 kB 00:00 2024-03-11T20:42:28.569 INFO:teuthology.orchestra.run.smithi129.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 592 kB/s | 44 kB 00:00 2024-03-11T20:42:28.619 INFO:teuthology.orchestra.run.smithi129.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 903 kB/s | 159 kB 00:00 2024-03-11T20:42:28.645 INFO:teuthology.orchestra.run.smithi129.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-11T20:42:28.670 INFO:teuthology.orchestra.run.smithi129.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.7 MB/s | 282 kB 00:00 2024-03-11T20:42:28.695 INFO:teuthology.orchestra.run.smithi129.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 552 kB/s | 41 kB 00:00 2024-03-11T20:42:28.978 INFO:teuthology.orchestra.run.smithi129.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 113 kB/s | 35 kB 00:00 2024-03-11T20:42:29.037 INFO:teuthology.orchestra.run.smithi129.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 130 kB/s | 44 kB 00:00 2024-03-11T20:42:29.204 INFO:teuthology.orchestra.run.smithi129.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 406 kB/s | 216 kB 00:00 2024-03-11T20:42:29.229 INFO:teuthology.orchestra.run.smithi129.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-11T20:42:29.288 INFO:teuthology.orchestra.run.smithi129.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 836 kB/s | 49 kB 00:00 2024-03-11T20:42:29.330 INFO:teuthology.orchestra.run.smithi129.stdout:(25/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 207 kB/s | 26 kB 00:00 2024-03-11T20:42:29.355 INFO:teuthology.orchestra.run.smithi129.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-11T20:42:29.489 INFO:teuthology.orchestra.run.smithi129.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-11T20:42:29.556 INFO:teuthology.orchestra.run.smithi129.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 8.5 MB/s | 4.4 MB 00:00 2024-03-11T20:42:29.689 INFO:teuthology.orchestra.run.smithi129.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-03-11T20:42:29.723 INFO:teuthology.orchestra.run.smithi129.stdout:(30/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.1 MB/s | 838 kB 00:00 2024-03-11T20:42:29.973 INFO:teuthology.orchestra.run.smithi129.stdout:(31/32): librbd1-19.0.0-1979.g2b0101eb.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-03-11T20:42:30.349 INFO:teuthology.orchestra.run.smithi129.stdout:(32/32): librados2-19.0.0-1979.g2b0101eb.el9.x8 4.3 MB/s | 3.4 MB 00:00 2024-03-11T20:42:30.352 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:42:30.352 INFO:teuthology.orchestra.run.smithi129.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-11T20:42:30.387 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:42:30.426 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:42:30.426 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:42:30.896 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-11T20:42:30.909 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:42:30.909 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:42:32.341 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:42:32.443 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-11T20:42:32.478 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-11T20:42:32.512 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-11T20:42:32.543 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-11T20:42:32.558 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-11T20:42:32.743 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-11T20:42:32.798 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-11T20:42:33.125 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-11T20:42:33.172 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-11T20:42:33.264 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-11T20:42:33.359 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 8/34 2024-03-11T20:42:33.398 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 9/34 2024-03-11T20:42:33.462 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 9/34 2024-03-11T20:42:33.476 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:42:33.478 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:42:33.478 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-11T20:42:33.478 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:42:33.478 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:42:33.478 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 9.8 M 2024-03-11T20:42:33.478 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrading: 2024-03-11T20:42:33.479 INFO:teuthology.orchestra.run.smithi157.stdout: librados2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.4 M 2024-03-11T20:42:33.479 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.1 M 2024-03-11T20:42:33.479 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-11T20:42:33.479 INFO:teuthology.orchestra.run.smithi157.stdout: boost-program-options 2024-03-11T20:42:33.479 INFO:teuthology.orchestra.run.smithi157.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T20:42:33.479 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 5.3 M 2024-03-11T20:42:33.479 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 20 M 2024-03-11T20:42:33.479 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 25 k 2024-03-11T20:42:33.479 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T20:42:33.479 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T20:42:33.479 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T20:42:33.479 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 745 k 2024-03-11T20:42:33.479 INFO:teuthology.orchestra.run.smithi157.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T20:42:33.479 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T20:42:33.480 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T20:42:33.480 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 501 k 2024-03-11T20:42:33.480 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T20:42:33.480 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 5.2 M 2024-03-11T20:42:33.480 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T20:42:33.480 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T20:42:33.480 INFO:teuthology.orchestra.run.smithi157.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T20:42:33.480 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse 2024-03-11T20:42:33.480 INFO:teuthology.orchestra.run.smithi157.stdout: x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 45 k 2024-03-11T20:42:33.480 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 131 k 2024-03-11T20:42:33.480 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 162 k 2024-03-11T20:42:33.480 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T20:42:33.480 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 319 k 2024-03-11T20:42:33.481 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 298 k 2024-03-11T20:42:33.481 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 99 k 2024-03-11T20:42:33.481 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T20:42:33.481 INFO:teuthology.orchestra.run.smithi157.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T20:42:33.481 INFO:teuthology.orchestra.run.smithi157.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T20:42:33.481 INFO:teuthology.orchestra.run.smithi157.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T20:42:33.481 INFO:teuthology.orchestra.run.smithi157.stdout:Installing weak dependencies: 2024-03-11T20:42:33.481 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T20:42:33.481 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:42:33.481 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:42:33.481 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:42:33.481 INFO:teuthology.orchestra.run.smithi157.stdout:Install 30 Packages 2024-03-11T20:42:33.481 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrade 2 Packages 2024-03-11T20:42:33.481 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:42:33.482 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 58 M 2024-03-11T20:42:33.482 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:42:33.498 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-11T20:42:33.534 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9. 11/34 2024-03-11T20:42:33.611 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 12/34 2024-03-11T20:42:33.642 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 13/34 2024-03-11T20:42:33.707 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 13/34 2024-03-11T20:42:33.744 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-11T20:42:33.909 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-11T20:42:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-11T20:42:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-11T20:42:34.025 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-11T20:42:34.056 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-11T20:42:34.088 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-11T20:42:34.414 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-11T20:42:34.440 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 22/34 2024-03-11T20:42:34.514 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 22/34 2024-03-11T20:42:34.545 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 23/34 2024-03-11T20:42:34.586 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-11T20:42:35.041 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-11T20:42:35.076 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 26/34 2024-03-11T20:42:35.155 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 26/34 2024-03-11T20:42:35.223 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 27/34 2024-03-11T20:42:35.253 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x8 28/34 2024-03-11T20:42:35.447 INFO:teuthology.orchestra.run.smithi157.stdout:(1/32): ceph-radosgw-19.0.0-1979.g2b0101eb.el9. 5.2 MB/s | 9.8 MB 00:01 2024-03-11T20:42:35.489 INFO:teuthology.orchestra.run.smithi157.stdout:(2/32): ceph-selinux-19.0.0-1979.g2b0101eb.el9. 608 kB/s | 25 kB 00:00 2024-03-11T20:42:35.589 INFO:teuthology.orchestra.run.smithi157.stdout:(3/32): libcephfs2-19.0.0-1979.g2b0101eb.el9.x8 7.3 MB/s | 745 kB 00:00 2024-03-11T20:42:35.656 INFO:teuthology.orchestra.run.smithi157.stdout:(4/32): libradosstriper1-19.0.0-1979.g2b0101eb. 7.3 MB/s | 501 kB 00:00 2024-03-11T20:42:35.840 INFO:teuthology.orchestra.run.smithi157.stdout:(5/32): ceph-base-19.0.0-1979.g2b0101eb.el9.x86 2.4 MB/s | 5.3 MB 00:02 2024-03-11T20:42:35.907 INFO:teuthology.orchestra.run.smithi157.stdout:(6/32): python3-ceph-argparse-19.0.0-1979.g2b01 674 kB/s | 45 kB 00:00 2024-03-11T20:42:35.974 INFO:teuthology.orchestra.run.smithi157.stdout:(7/32): python3-ceph-common-19.0.0-1979.g2b0101 1.9 MB/s | 131 kB 00:00 2024-03-11T20:42:36.057 INFO:teuthology.orchestra.run.smithi157.stdout:(8/32): python3-cephfs-19.0.0-1979.g2b0101eb.el 1.9 MB/s | 162 kB 00:00 2024-03-11T20:42:36.124 INFO:teuthology.orchestra.run.smithi157.stdout:(9/32): librgw2-19.0.0-1979.g2b0101eb.el9.x86_6 11 MB/s | 5.2 MB 00:00 2024-03-11T20:42:36.183 INFO:teuthology.orchestra.run.smithi157.stdout:(10/32): python3-rbd-19.0.0-1979.g2b0101eb.el9. 5.0 MB/s | 298 kB 00:00 2024-03-11T20:42:36.208 INFO:teuthology.orchestra.run.smithi157.stdout:(11/32): python3-rados-19.0.0-1979.g2b0101eb.el 2.1 MB/s | 319 kB 00:00 2024-03-11T20:42:36.233 INFO:teuthology.orchestra.run.smithi157.stdout:(12/32): python3-rgw-19.0.0-1979.g2b0101eb.el9. 1.9 MB/s | 99 kB 00:00 2024-03-11T20:42:36.367 INFO:teuthology.orchestra.run.smithi157.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 666 kB/s | 106 kB 00:00 2024-03-11T20:42:36.393 INFO:teuthology.orchestra.run.smithi157.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-11T20:42:36.418 INFO:teuthology.orchestra.run.smithi157.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-11T20:42:36.476 INFO:teuthology.orchestra.run.smithi157.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-11T20:42:36.502 INFO:teuthology.orchestra.run.smithi157.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-11T20:42:36.527 INFO:teuthology.orchestra.run.smithi157.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 824 kB/s | 41 kB 00:00 2024-03-11T20:42:36.636 INFO:teuthology.orchestra.run.smithi157.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 322 kB/s | 35 kB 00:00 2024-03-11T20:42:36.702 INFO:teuthology.orchestra.run.smithi157.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.1 MB/s | 216 kB 00:00 2024-03-11T20:42:36.728 INFO:teuthology.orchestra.run.smithi157.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 483 kB/s | 44 kB 00:00 2024-03-11T20:42:37.236 INFO:teuthology.orchestra.run.smithi157.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 578 kB/s | 309 kB 00:00 2024-03-11T20:42:37.337 INFO:teuthology.orchestra.run.smithi157.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-03-11T20:42:37.340 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-11T20:42:37.383 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-11T20:42:37.470 INFO:teuthology.orchestra.run.smithi157.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 365 kB/s | 49 kB 00:00 2024-03-11T20:42:37.596 INFO:teuthology.orchestra.run.smithi157.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 539 kB/s | 67 kB 00:00 2024-03-11T20:42:37.934 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-11T20:42:37.970 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 30/34 2024-03-11T20:42:38.013 INFO:teuthology.orchestra.run.smithi157.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 4.4 MB 00:01 2024-03-11T20:42:38.052 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 30/34 2024-03-11T20:42:38.052 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T20:42:38.052 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T20:42:38.052 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:42:38.054 INFO:teuthology.orchestra.run.smithi157.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.8 MB/s | 838 kB 00:00 2024-03-11T20:42:38.088 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-11T20:42:38.130 INFO:teuthology.orchestra.run.smithi157.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-03-11T20:42:38.814 INFO:teuthology.orchestra.run.smithi157.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2024-03-11T20:42:39.589 INFO:teuthology.orchestra.run.smithi157.stdout:(30/32): librbd1-19.0.0-1979.g2b0101eb.el9.x86_ 4.1 MB/s | 3.1 MB 00:00 2024-03-11T20:42:39.681 INFO:teuthology.orchestra.run.smithi157.stdout:(31/32): librados2-19.0.0-1979.g2b0101eb.el9.x8 2.2 MB/s | 3.4 MB 00:01 2024-03-11T20:42:40.498 INFO:teuthology.orchestra.run.smithi157.stdout:(32/32): ceph-common-19.0.0-1979.g2b0101eb.el9. 2.9 MB/s | 20 MB 00:06 2024-03-11T20:42:40.502 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:42:40.502 INFO:teuthology.orchestra.run.smithi157.stdout:Total 8.3 MB/s | 58 MB 00:07 2024-03-11T20:42:40.537 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:42:40.575 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:42:40.575 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:42:41.050 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:42:41.050 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:42:42.462 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:42:42.550 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-11T20:42:42.577 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-11T20:42:42.618 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-11T20:42:42.642 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-11T20:42:42.658 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-11T20:42:42.863 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-11T20:42:42.923 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-11T20:42:43.256 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-11T20:42:43.313 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-11T20:42:43.388 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-11T20:42:43.478 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 8/34 2024-03-11T20:42:43.531 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 9/34 2024-03-11T20:42:43.603 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 9/34 2024-03-11T20:42:43.631 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-11T20:42:43.687 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9. 11/34 2024-03-11T20:42:43.762 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 12/34 2024-03-11T20:42:43.783 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 13/34 2024-03-11T20:42:43.846 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 13/34 2024-03-11T20:42:43.882 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-11T20:42:44.002 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-11T20:42:44.042 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-11T20:42:44.082 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-11T20:42:44.115 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-11T20:42:44.143 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-11T20:42:44.176 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-11T20:42:44.499 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-11T20:42:44.532 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 22/34 2024-03-11T20:42:44.604 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 22/34 2024-03-11T20:42:44.645 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 23/34 2024-03-11T20:42:44.697 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-11T20:42:45.157 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-11T20:42:45.184 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 26/34 2024-03-11T20:42:45.255 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 26/34 2024-03-11T20:42:45.324 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 27/34 2024-03-11T20:42:45.353 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x8 28/34 2024-03-11T20:42:47.509 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-11T20:42:47.558 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-11T20:42:47.839 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-11T20:42:47.839 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /sys 2024-03-11T20:42:47.839 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /proc 2024-03-11T20:42:47.840 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /mnt 2024-03-11T20:42:47.840 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /var/tmp 2024-03-11T20:42:47.840 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /home 2024-03-11T20:42:47.840 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /root 2024-03-11T20:42:47.840 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /tmp 2024-03-11T20:42:47.840 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:42:48.122 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-11T20:42:48.162 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 30/34 2024-03-11T20:42:48.242 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 30/34 2024-03-11T20:42:48.242 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T20:42:48.242 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T20:42:48.242 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:42:48.278 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-11T20:42:48.766 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/34 2024-03-11T20:42:48.791 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/34 2024-03-11T20:42:48.791 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T20:42:48.791 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T20:42:48.791 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T20:42:48.791 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T20:42:48.791 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:42:48.793 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-11T20:42:48.860 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-11T20:42:48.860 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T20:42:50.034 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T20:42:50.034 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/34 2024-03-11T20:42:50.034 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/34 2024-03-11T20:42:50.034 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 3/34 2024-03-11T20:42:50.035 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 4/34 2024-03-11T20:42:50.035 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/34 2024-03-11T20:42:50.035 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 6/34 2024-03-11T20:42:50.035 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-11T20:42:50.035 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9. 8/34 2024-03-11T20:42:50.035 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x8 9/34 2024-03-11T20:42:50.035 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 10/34 2024-03-11T20:42:50.035 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 11/34 2024-03-11T20:42:50.035 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 12/34 2024-03-11T20:42:50.035 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 13/34 2024-03-11T20:42:50.035 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-11T20:42:50.035 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-11T20:42:50.035 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-11T20:42:50.036 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-11T20:42:50.036 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-11T20:42:50.036 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-11T20:42:50.036 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-11T20:42:50.036 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-11T20:42:50.036 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-11T20:42:50.036 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-11T20:42:50.036 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-11T20:42:50.036 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-11T20:42:50.036 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-11T20:42:50.036 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-11T20:42:50.036 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-11T20:42:50.036 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-11T20:42:50.037 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-11T20:42:50.037 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-11T20:42:50.037 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-11T20:42:50.037 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/34 2024-03-11T20:42:51.017 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T20:42:51.018 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:42:51.018 INFO:teuthology.orchestra.run.smithi129.stdout:Upgraded: 2024-03-11T20:42:51.018 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:42:51.018 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:42:51.018 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:42:51.018 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T20:42:51.018 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:42:51.018 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:42:51.018 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:42:51.018 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:42:51.018 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T20:42:51.018 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T20:42:51.019 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T20:42:51.019 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:42:51.019 INFO:teuthology.orchestra.run.smithi129.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T20:42:51.019 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T20:42:51.019 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T20:42:51.019 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:42:51.019 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T20:42:51.019 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:42:51.019 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T20:42:51.019 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T20:42:51.019 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T20:42:51.019 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:42:51.019 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:42:51.019 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:42:51.020 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T20:42:51.020 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:42:51.020 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:42:51.020 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:42:51.020 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T20:42:51.020 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T20:42:51.020 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T20:42:51.020 INFO:teuthology.orchestra.run.smithi129.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T20:42:51.020 INFO:teuthology.orchestra.run.smithi129.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T20:42:51.020 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:42:51.021 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:42:51.362 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-test 2024-03-11T20:42:51.914 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:28 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:42:52.062 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:42:52.063 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:42:52.063 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-11T20:42:52.063 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:42:52.063 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:42:52.063 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 48 M 2024-03-11T20:42:52.064 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-11T20:42:52.064 INFO:teuthology.orchestra.run.smithi129.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T20:42:52.064 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 163 k 2024-03-11T20:42:52.064 INFO:teuthology.orchestra.run.smithi129.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T20:42:52.064 INFO:teuthology.orchestra.run.smithi129.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T20:42:52.064 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T20:42:52.064 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:42:52.064 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:42:52.064 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:42:52.064 INFO:teuthology.orchestra.run.smithi129.stdout:Install 6 Packages 2024-03-11T20:42:52.064 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:42:52.065 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 48 M 2024-03-11T20:42:52.065 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 205 M 2024-03-11T20:42:52.065 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:42:52.296 INFO:teuthology.orchestra.run.smithi129.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-11T20:42:52.355 INFO:teuthology.orchestra.run.smithi129.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-11T20:42:52.380 INFO:teuthology.orchestra.run.smithi129.stdout:(3/6): libcephsqlite-19.0.0-1979.g2b0101eb.el9. 605 kB/s | 163 kB 00:00 2024-03-11T20:42:52.397 INFO:teuthology.orchestra.run.smithi129.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-11T20:42:53.347 INFO:teuthology.orchestra.run.smithi129.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 194 kB/s | 187 kB 00:00 2024-03-11T20:42:54.922 INFO:teuthology.orchestra.run.smithi129.stdout:(6/6): ceph-test-19.0.0-1979.g2b0101eb.el9.x86_ 17 MB/s | 48 MB 00:02 2024-03-11T20:42:54.926 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:42:54.926 INFO:teuthology.orchestra.run.smithi129.stdout:Total 17 MB/s | 48 MB 00:02 2024-03-11T20:42:54.941 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:42:54.962 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:42:54.962 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:42:55.344 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:42:55.344 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:42:55.916 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:42:55.989 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-11T20:42:56.042 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-11T20:42:56.083 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T20:42:56.123 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-11T20:42:56.150 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/6 2024-03-11T20:42:58.009 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-11T20:42:58.009 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /sys 2024-03-11T20:42:58.010 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /proc 2024-03-11T20:42:58.010 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /mnt 2024-03-11T20:42:58.010 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /var/tmp 2024-03-11T20:42:58.010 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /home 2024-03-11T20:42:58.010 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /root 2024-03-11T20:42:58.010 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /tmp 2024-03-11T20:42:58.010 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:42:58.925 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/34 2024-03-11T20:42:58.950 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/34 2024-03-11T20:42:58.950 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T20:42:58.950 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T20:42:58.950 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T20:42:58.950 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T20:42:58.950 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:42:58.952 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-11T20:42:59.035 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-11T20:42:59.036 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T20:43:00.119 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T20:43:00.119 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/34 2024-03-11T20:43:00.119 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/34 2024-03-11T20:43:00.119 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 3/34 2024-03-11T20:43:00.119 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 4/34 2024-03-11T20:43:00.119 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/34 2024-03-11T20:43:00.119 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 6/34 2024-03-11T20:43:00.119 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-11T20:43:00.119 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9. 8/34 2024-03-11T20:43:00.119 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x8 9/34 2024-03-11T20:43:00.119 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 10/34 2024-03-11T20:43:00.119 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 11/34 2024-03-11T20:43:00.120 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 12/34 2024-03-11T20:43:00.120 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 13/34 2024-03-11T20:43:00.120 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-11T20:43:00.120 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-11T20:43:00.120 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-11T20:43:00.120 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-11T20:43:00.120 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-11T20:43:00.120 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-11T20:43:00.120 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-11T20:43:00.120 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-11T20:43:00.120 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-11T20:43:00.120 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-11T20:43:00.120 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-11T20:43:00.120 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-11T20:43:00.121 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-11T20:43:00.121 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-11T20:43:00.122 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-11T20:43:00.122 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-11T20:43:00.122 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-11T20:43:00.122 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-11T20:43:00.122 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-11T20:43:00.122 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/34 2024-03-11T20:43:00.671 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/6 2024-03-11T20:43:00.750 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 6/6 2024-03-11T20:43:01.374 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T20:43:01.374 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:01.375 INFO:teuthology.orchestra.run.smithi157.stdout:Upgraded: 2024-03-11T20:43:01.375 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:01.375 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:01.375 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:43:01.375 INFO:teuthology.orchestra.run.smithi157.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T20:43:01.375 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:01.375 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:01.375 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:01.375 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:01.375 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T20:43:01.375 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T20:43:01.376 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T20:43:01.376 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:01.376 INFO:teuthology.orchestra.run.smithi157.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T20:43:01.376 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T20:43:01.376 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T20:43:01.376 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:01.376 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T20:43:01.376 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:01.376 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T20:43:01.376 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T20:43:01.376 INFO:teuthology.orchestra.run.smithi157.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T20:43:01.376 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:01.376 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:01.377 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:01.377 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T20:43:01.377 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:01.377 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:01.377 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:01.377 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T20:43:01.377 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T20:43:01.377 INFO:teuthology.orchestra.run.smithi157.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T20:43:01.377 INFO:teuthology.orchestra.run.smithi157.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T20:43:01.377 INFO:teuthology.orchestra.run.smithi157.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T20:43:01.377 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:01.377 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:43:01.696 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-test 2024-03-11T20:43:02.277 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:32 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:43:02.434 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:43:02.434 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:43:02.435 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-11T20:43:02.435 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:43:02.435 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:43:02.435 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 48 M 2024-03-11T20:43:02.435 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-11T20:43:02.435 INFO:teuthology.orchestra.run.smithi157.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T20:43:02.435 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 163 k 2024-03-11T20:43:02.435 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T20:43:02.435 INFO:teuthology.orchestra.run.smithi157.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T20:43:02.435 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T20:43:02.436 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:02.436 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:43:02.436 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:43:02.436 INFO:teuthology.orchestra.run.smithi157.stdout:Install 6 Packages 2024-03-11T20:43:02.436 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:02.436 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 48 M 2024-03-11T20:43:02.437 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 205 M 2024-03-11T20:43:02.437 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:43:02.781 INFO:teuthology.orchestra.run.smithi157.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-11T20:43:02.839 INFO:teuthology.orchestra.run.smithi157.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-11T20:43:02.873 INFO:teuthology.orchestra.run.smithi157.stdout:(3/6): libcephsqlite-19.0.0-1979.g2b0101eb.el9. 585 kB/s | 163 kB 00:00 2024-03-11T20:43:02.898 INFO:teuthology.orchestra.run.smithi157.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-11T20:43:03.240 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 6/6 2024-03-11T20:43:03.240 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/6 2024-03-11T20:43:03.240 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/6 2024-03-11T20:43:03.240 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T20:43:03.240 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-11T20:43:03.240 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-11T20:43:03.448 INFO:teuthology.orchestra.run.smithi157.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 325 kB/s | 187 kB 00:00 2024-03-11T20:43:03.653 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-11T20:43:03.653 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:03.654 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:43:03.654 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:03.654 INFO:teuthology.orchestra.run.smithi129.stdout: jq-1.6-16.el9.x86_64 2024-03-11T20:43:03.654 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:03.654 INFO:teuthology.orchestra.run.smithi129.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T20:43:03.654 INFO:teuthology.orchestra.run.smithi129.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T20:43:03.654 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T20:43:03.654 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:03.654 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:43:03.871 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph 2024-03-11T20:43:04.465 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:41 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:43:04.627 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:43:04.629 INFO:teuthology.orchestra.run.smithi129.stdout:========================================================================================= 2024-03-11T20:43:04.629 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-11T20:43:04.629 INFO:teuthology.orchestra.run.smithi129.stdout:========================================================================================= 2024-03-11T20:43:04.629 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:43:04.629 INFO:teuthology.orchestra.run.smithi129.stdout: ceph x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 6.6 k 2024-03-11T20:43:04.630 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-11T20:43:04.630 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 2.1 M 2024-03-11T20:43:04.630 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 1.5 M 2024-03-11T20:43:04.630 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 248 k 2024-03-11T20:43:04.630 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 4.7 M 2024-03-11T20:43:04.630 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 17 M 2024-03-11T20:43:04.630 INFO:teuthology.orchestra.run.smithi129.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T20:43:04.630 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T20:43:04.630 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T20:43:04.631 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T20:43:04.631 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T20:43:04.631 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T20:43:04.631 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T20:43:04.631 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T20:43:04.631 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T20:43:04.631 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T20:43:04.631 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T20:43:04.631 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T20:43:04.631 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T20:43:04.631 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T20:43:04.632 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T20:43:04.632 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T20:43:04.632 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T20:43:04.632 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T20:43:04.632 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T20:43:04.632 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T20:43:04.632 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T20:43:04.632 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T20:43:04.632 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T20:43:04.632 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T20:43:04.632 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T20:43:04.632 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T20:43:04.633 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T20:43:04.633 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T20:43:04.633 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T20:43:04.633 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T20:43:04.633 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:04.633 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:43:04.633 INFO:teuthology.orchestra.run.smithi129.stdout:========================================================================================= 2024-03-11T20:43:04.633 INFO:teuthology.orchestra.run.smithi129.stdout:Install 36 Packages 2024-03-11T20:43:04.633 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:04.634 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 30 M 2024-03-11T20:43:04.634 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 107 M 2024-03-11T20:43:04.634 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:43:05.278 INFO:teuthology.orchestra.run.smithi129.stdout:(1/36): ceph-19.0.0-1979.g2b0101eb.el9.x86_64.r 27 kB/s | 6.6 kB 00:00 2024-03-11T20:43:06.444 INFO:teuthology.orchestra.run.smithi129.stdout:(2/36): ceph-mds-19.0.0-1979.g2b0101eb.el9.x86_ 1.5 MB/s | 2.1 MB 00:01 2024-03-11T20:43:06.457 INFO:teuthology.orchestra.run.smithi157.stdout:(6/6): ceph-test-19.0.0-1979.g2b0101eb.el9.x86_ 12 MB/s | 48 MB 00:03 2024-03-11T20:43:06.460 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:43:06.460 INFO:teuthology.orchestra.run.smithi157.stdout:Total 12 MB/s | 48 MB 00:04 2024-03-11T20:43:06.474 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:43:06.496 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:43:06.496 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:43:06.787 INFO:teuthology.orchestra.run.smithi129.stdout:(3/36): ceph-mgr-19.0.0-1979.g2b0101eb.el9.x86_ 870 kB/s | 1.5 MB 00:01 2024-03-11T20:43:06.885 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:43:06.885 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:43:06.962 INFO:teuthology.orchestra.run.smithi129.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1979.g2b01 1.4 MB/s | 248 kB 00:00 2024-03-11T20:43:07.103 INFO:teuthology.orchestra.run.smithi129.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-11T20:43:07.146 INFO:teuthology.orchestra.run.smithi129.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-11T20:43:07.179 INFO:teuthology.orchestra.run.smithi129.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2024-03-11T20:43:07.213 INFO:teuthology.orchestra.run.smithi129.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-11T20:43:07.247 INFO:teuthology.orchestra.run.smithi129.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-11T20:43:07.305 INFO:teuthology.orchestra.run.smithi129.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-11T20:43:07.406 INFO:teuthology.orchestra.run.smithi129.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-11T20:43:07.447 INFO:teuthology.orchestra.run.smithi129.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-11T20:43:07.476 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:43:07.481 INFO:teuthology.orchestra.run.smithi129.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-11T20:43:07.523 INFO:teuthology.orchestra.run.smithi129.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-11T20:43:07.557 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-11T20:43:07.596 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-11T20:43:07.646 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T20:43:07.690 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-11T20:43:07.718 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/6 2024-03-11T20:43:07.723 INFO:teuthology.orchestra.run.smithi129.stdout:(15/36): ceph-mon-19.0.0-1979.g2b0101eb.el9.x86 1.9 MB/s | 4.7 MB 00:02 2024-03-11T20:43:07.791 INFO:teuthology.orchestra.run.smithi129.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 468 kB/s | 125 kB 00:00 2024-03-11T20:43:07.816 INFO:teuthology.orchestra.run.smithi129.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-11T20:43:08.166 INFO:teuthology.orchestra.run.smithi129.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2024-03-11T20:43:08.208 INFO:teuthology.orchestra.run.smithi129.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 362 kB/s | 151 kB 00:00 2024-03-11T20:43:08.409 INFO:teuthology.orchestra.run.smithi129.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-03-11T20:43:08.434 INFO:teuthology.orchestra.run.smithi129.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 643 kB/s | 172 kB 00:00 2024-03-11T20:43:08.517 INFO:teuthology.orchestra.run.smithi129.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 98 kB/s | 11 kB 00:00 2024-03-11T20:43:08.543 INFO:teuthology.orchestra.run.smithi129.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-03-11T20:43:08.618 INFO:teuthology.orchestra.run.smithi129.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-11T20:43:08.643 INFO:teuthology.orchestra.run.smithi129.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-03-11T20:43:08.718 INFO:teuthology.orchestra.run.smithi129.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-03-11T20:43:08.752 INFO:teuthology.orchestra.run.smithi129.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 427 kB/s | 46 kB 00:00 2024-03-11T20:43:08.827 INFO:teuthology.orchestra.run.smithi129.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 726 kB/s | 79 kB 00:00 2024-03-11T20:43:08.936 INFO:teuthology.orchestra.run.smithi129.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-03-11T20:43:08.962 INFO:teuthology.orchestra.run.smithi129.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-11T20:43:09.045 INFO:teuthology.orchestra.run.smithi129.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 829 kB/s | 90 kB 00:00 2024-03-11T20:43:09.087 INFO:teuthology.orchestra.run.smithi129.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 286 kB/s | 36 kB 00:00 2024-03-11T20:43:09.179 INFO:teuthology.orchestra.run.smithi129.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-11T20:43:09.213 INFO:teuthology.orchestra.run.smithi129.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-03-11T20:43:09.313 INFO:teuthology.orchestra.run.smithi129.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 150 kB/s | 20 kB 00:00 2024-03-11T20:43:09.938 INFO:teuthology.orchestra.run.smithi129.stdout:(36/36): ceph-osd-19.0.0-1979.g2b0101eb.el9.x86 4.8 MB/s | 17 MB 00:03 2024-03-11T20:43:09.942 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:43:09.942 INFO:teuthology.orchestra.run.smithi129.stdout:Total 5.6 MB/s | 30 MB 00:05 2024-03-11T20:43:09.967 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:43:10.010 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:43:10.011 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:43:10.438 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:43:10.439 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:43:11.756 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:43:11.824 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-11T20:43:11.867 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-11T20:43:11.894 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-11T20:43:11.965 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-11T20:43:12.009 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-11T20:43:12.042 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-11T20:43:12.133 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-11T20:43:12.203 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-11T20:43:12.233 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/6 2024-03-11T20:43:12.262 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-11T20:43:12.308 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 6/6 2024-03-11T20:43:12.333 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-11T20:43:12.396 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-11T20:43:12.434 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-11T20:43:12.483 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-11T20:43:12.538 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-11T20:43:12.727 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-11T20:43:12.780 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-11T20:43:12.814 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-11T20:43:12.853 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-11T20:43:12.883 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-11T20:43:12.959 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-11T20:43:12.994 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-11T20:43:13.027 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-11T20:43:13.068 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-11T20:43:13.102 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-11T20:43:13.137 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-11T20:43:13.225 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-11T20:43:13.277 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-11T20:43:13.349 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-11T20:43:13.484 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-11T20:43:13.612 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-11T20:43:13.777 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9. 31/36 2024-03-11T20:43:13.807 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/36 2024-03-11T20:43:13.832 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/36 2024-03-11T20:43:13.832 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T20:43:13.832 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T20:43:13.832 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T20:43:13.832 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T20:43:13.832 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:14.761 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 6/6 2024-03-11T20:43:14.761 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/6 2024-03-11T20:43:14.762 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/6 2024-03-11T20:43:14.762 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T20:43:14.762 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-11T20:43:14.762 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-11T20:43:15.211 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-11T20:43:15.211 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:15.211 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:43:15.211 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:15.211 INFO:teuthology.orchestra.run.smithi157.stdout: jq-1.6-16.el9.x86_64 2024-03-11T20:43:15.211 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:15.212 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T20:43:15.212 INFO:teuthology.orchestra.run.smithi157.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T20:43:15.212 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T20:43:15.212 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:15.212 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:43:15.301 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/36 2024-03-11T20:43:15.327 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/36 2024-03-11T20:43:15.327 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T20:43:15.327 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T20:43:15.327 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T20:43:15.327 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T20:43:15.327 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:15.443 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph 2024-03-11T20:43:15.767 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 34/36 2024-03-11T20:43:15.789 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 34/36 2024-03-11T20:43:15.790 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T20:43:15.790 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T20:43:15.790 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T20:43:15.790 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T20:43:15.790 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:15.993 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 35/36 2024-03-11T20:43:16.015 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 35/36 2024-03-11T20:43:16.015 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T20:43:16.015 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T20:43:16.015 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T20:43:16.015 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T20:43:16.015 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:16.031 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:46 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:43:16.079 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 36/36 2024-03-11T20:43:16.202 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:43:16.205 INFO:teuthology.orchestra.run.smithi157.stdout:========================================================================================= 2024-03-11T20:43:16.205 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-11T20:43:16.205 INFO:teuthology.orchestra.run.smithi157.stdout:========================================================================================= 2024-03-11T20:43:16.205 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:43:16.205 INFO:teuthology.orchestra.run.smithi157.stdout: ceph x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 6.6 k 2024-03-11T20:43:16.205 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-11T20:43:16.205 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 2.1 M 2024-03-11T20:43:16.205 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 1.5 M 2024-03-11T20:43:16.205 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 248 k 2024-03-11T20:43:16.205 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 4.7 M 2024-03-11T20:43:16.206 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 17 M 2024-03-11T20:43:16.206 INFO:teuthology.orchestra.run.smithi157.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T20:43:16.206 INFO:teuthology.orchestra.run.smithi157.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T20:43:16.206 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T20:43:16.206 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T20:43:16.206 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T20:43:16.206 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T20:43:16.206 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T20:43:16.206 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T20:43:16.206 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T20:43:16.206 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T20:43:16.206 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T20:43:16.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T20:43:16.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T20:43:16.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T20:43:16.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T20:43:16.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T20:43:16.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T20:43:16.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T20:43:16.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T20:43:16.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T20:43:16.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T20:43:16.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T20:43:16.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T20:43:16.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T20:43:16.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T20:43:16.208 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T20:43:16.208 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T20:43:16.208 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T20:43:16.208 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T20:43:16.208 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T20:43:16.208 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:16.208 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:43:16.208 INFO:teuthology.orchestra.run.smithi157.stdout:========================================================================================= 2024-03-11T20:43:16.208 INFO:teuthology.orchestra.run.smithi157.stdout:Install 36 Packages 2024-03-11T20:43:16.208 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:16.209 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 30 M 2024-03-11T20:43:16.209 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 107 M 2024-03-11T20:43:16.209 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:43:16.667 INFO:teuthology.orchestra.run.smithi157.stdout:(1/36): ceph-19.0.0-1979.g2b0101eb.el9.x86_64.r 31 kB/s | 6.6 kB 00:00 2024-03-11T20:43:16.850 INFO:teuthology.orchestra.run.smithi157.stdout:(2/36): ceph-mgr-19.0.0-1979.g2b0101eb.el9.x86_ 3.8 MB/s | 1.5 MB 00:00 2024-03-11T20:43:17.176 INFO:teuthology.orchestra.run.smithi157.stdout:(3/36): ceph-mds-19.0.0-1979.g2b0101eb.el9.x86_ 2.9 MB/s | 2.1 MB 00:00 2024-03-11T20:43:17.300 INFO:teuthology.orchestra.run.smithi157.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1979.g2b01 1.9 MB/s | 248 kB 00:00 2024-03-11T20:43:17.435 INFO:teuthology.orchestra.run.smithi157.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-11T20:43:17.477 INFO:teuthology.orchestra.run.smithi157.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-11T20:43:17.519 INFO:teuthology.orchestra.run.smithi157.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 755 kB/s | 32 kB 00:00 2024-03-11T20:43:17.577 INFO:teuthology.orchestra.run.smithi157.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 812 kB/s | 47 kB 00:00 2024-03-11T20:43:17.619 INFO:teuthology.orchestra.run.smithi157.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 928 kB/s | 39 kB 00:00 2024-03-11T20:43:17.670 INFO:teuthology.orchestra.run.smithi157.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-11T20:43:17.745 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 36/36 2024-03-11T20:43:17.745 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/36 2024-03-11T20:43:17.745 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/36 2024-03-11T20:43:17.745 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 3/36 2024-03-11T20:43:17.745 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 4/36 2024-03-11T20:43:17.745 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/36 2024-03-11T20:43:17.746 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9. 6/36 2024-03-11T20:43:17.746 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-11T20:43:17.746 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-11T20:43:17.746 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-11T20:43:17.746 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-11T20:43:17.746 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-11T20:43:17.746 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-11T20:43:17.746 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-11T20:43:17.746 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-11T20:43:17.746 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-11T20:43:17.746 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-11T20:43:17.746 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-11T20:43:17.746 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-11T20:43:17.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-11T20:43:17.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-11T20:43:17.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-11T20:43:17.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-11T20:43:17.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-11T20:43:17.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-11T20:43:17.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-11T20:43:17.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-11T20:43:17.747 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-11T20:43:17.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-11T20:43:17.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-11T20:43:17.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-11T20:43:17.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-11T20:43:17.748 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-11T20:43:17.749 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-11T20:43:17.749 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-11T20:43:17.749 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-11T20:43:17.753 INFO:teuthology.orchestra.run.smithi157.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-11T20:43:17.920 INFO:teuthology.orchestra.run.smithi157.stdout:(12/36): ceph-osd-19.0.0-1979.g2b0101eb.el9.x86 16 MB/s | 17 MB 00:01 2024-03-11T20:43:17.945 INFO:teuthology.orchestra.run.smithi157.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 535 kB/s | 103 kB 00:00 2024-03-11T20:43:17.971 INFO:teuthology.orchestra.run.smithi157.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-11T20:43:18.013 INFO:teuthology.orchestra.run.smithi157.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.7 MB/s | 187 kB 00:00 2024-03-11T20:43:18.080 INFO:teuthology.orchestra.run.smithi157.stdout:(16/36): ceph-mon-19.0.0-1979.g2b0101eb.el9.x86 3.3 MB/s | 4.7 MB 00:01 2024-03-11T20:43:18.205 INFO:teuthology.orchestra.run.smithi157.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 535 kB/s | 125 kB 00:00 2024-03-11T20:43:18.280 INFO:teuthology.orchestra.run.smithi157.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 83 kB/s | 22 kB 00:00 2024-03-11T20:43:18.314 INFO:teuthology.orchestra.run.smithi157.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 646 kB/s | 151 kB 00:00 2024-03-11T20:43:18.414 INFO:teuthology.orchestra.run.smithi157.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 208 kB/s | 43 kB 00:00 2024-03-11T20:43:18.466 INFO:teuthology.orchestra.run.smithi157.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-03-11T20:43:18.506 INFO:teuthology.orchestra.run.smithi157.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 425 kB/s | 18 kB 00:00 2024-03-11T20:43:18.531 INFO:teuthology.orchestra.run.smithi157.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 686 kB/s | 172 kB 00:00 2024-03-11T20:43:18.557 INFO:teuthology.orchestra.run.smithi157.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-03-11T20:43:18.582 INFO:teuthology.orchestra.run.smithi157.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-11T20:43:18.607 INFO:teuthology.orchestra.run.smithi157.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-03-11T20:43:18.633 INFO:teuthology.orchestra.run.smithi157.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-03-11T20:43:18.666 INFO:teuthology.orchestra.run.smithi157.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 791 kB/s | 46 kB 00:00 2024-03-11T20:43:18.691 INFO:teuthology.orchestra.run.smithi157.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 941 kB/s | 79 kB 00:00 2024-03-11T20:43:18.775 INFO:teuthology.orchestra.run.smithi157.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-03-11T20:43:18.800 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-11T20:43:18.800 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:18.800 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:43:18.800 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:18.800 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:18.800 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:18.800 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:43:18.800 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:18.800 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:18.800 INFO:teuthology.orchestra.run.smithi129.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T20:43:18.800 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T20:43:18.800 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T20:43:18.800 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T20:43:18.800 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T20:43:18.801 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T20:43:18.801 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T20:43:18.801 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T20:43:18.801 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T20:43:18.801 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T20:43:18.801 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T20:43:18.801 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T20:43:18.801 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T20:43:18.801 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T20:43:18.801 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T20:43:18.801 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T20:43:18.801 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T20:43:18.801 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T20:43:18.801 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T20:43:18.802 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T20:43:18.802 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T20:43:18.802 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T20:43:18.802 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T20:43:18.802 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T20:43:18.802 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T20:43:18.802 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T20:43:18.802 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T20:43:18.802 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T20:43:18.802 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T20:43:18.802 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T20:43:18.802 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:18.802 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:43:18.858 INFO:teuthology.orchestra.run.smithi157.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 429 kB/s | 36 kB 00:00 2024-03-11T20:43:18.884 INFO:teuthology.orchestra.run.smithi157.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 468 kB/s | 90 kB 00:00 2024-03-11T20:43:18.917 INFO:teuthology.orchestra.run.smithi157.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 957 kB/s | 272 kB 00:00 2024-03-11T20:43:18.984 INFO:teuthology.orchestra.run.smithi157.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2024-03-11T20:43:19.083 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-base 2024-03-11T20:43:19.426 INFO:teuthology.orchestra.run.smithi157.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 788 kB/s | 427 kB 00:00 2024-03-11T20:43:19.643 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:56 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:43:19.739 INFO:teuthology.orchestra.run.smithi129.stdout:Package ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:43:19.760 INFO:teuthology.orchestra.run.smithi157.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 255 kB/s | 230 kB 00:00 2024-03-11T20:43:19.765 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:43:19.765 INFO:teuthology.orchestra.run.smithi157.stdout:Total 8.4 MB/s | 30 MB 00:03 2024-03-11T20:43:19.789 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:43:19.790 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:43:19.791 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-11T20:43:19.791 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:43:19.833 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:43:19.833 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:43:19.844 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install cephadm 2024-03-11T20:43:20.248 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:43:20.249 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:43:20.406 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:57 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:43:20.554 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:43:20.554 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:43:20.555 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-11T20:43:20.555 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:43:20.555 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:43:20.555 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 748 k 2024-03-11T20:43:20.555 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:20.555 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:43:20.555 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:43:20.555 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-03-11T20:43:20.555 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:20.555 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 748 k 2024-03-11T20:43:20.555 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 755 k 2024-03-11T20:43:20.555 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:43:21.323 INFO:teuthology.orchestra.run.smithi129.stdout:cephadm-19.0.0-1979.g2b0101eb.el9.noarch.rpm 975 kB/s | 748 kB 00:00 2024-03-11T20:43:21.324 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:43:21.324 INFO:teuthology.orchestra.run.smithi129.stdout:Total 973 kB/s | 748 kB 00:00 2024-03-11T20:43:21.324 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:43:21.329 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:43:21.329 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:43:21.335 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:43:21.335 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:43:21.412 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:43:21.575 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:43:21.646 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-11T20:43:21.679 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-11T20:43:21.706 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-11T20:43:21.777 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-11T20:43:21.830 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-11T20:43:21.863 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-11T20:43:21.864 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-11T20:43:21.953 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-11T20:43:21.955 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-11T20:43:22.032 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-11T20:43:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-11T20:43:22.146 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-11T20:43:22.199 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-11T20:43:22.238 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-11T20:43:22.286 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-11T20:43:22.342 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-11T20:43:22.348 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-11T20:43:22.531 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-11T20:43:22.592 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-11T20:43:22.633 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-11T20:43:22.633 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-11T20:43:22.633 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:22.633 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:43:22.633 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:43:22.633 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:22.633 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:43:22.665 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-11T20:43:22.696 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-11T20:43:22.771 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-11T20:43:22.806 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-11T20:43:22.839 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-11T20:43:22.871 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-immutable-object-cache 2024-03-11T20:43:22.879 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-11T20:43:22.923 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-11T20:43:22.966 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-11T20:43:23.055 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-11T20:43:23.097 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-11T20:43:23.168 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-11T20:43:23.305 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-11T20:43:23.430 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-11T20:43:23.450 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:00 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:43:23.597 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:43:23.598 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:43:23.598 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-03-11T20:43:23.598 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:43:23.598 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:43:23.598 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 145 k 2024-03-11T20:43:23.598 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:23.598 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:43:23.598 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:43:23.599 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-03-11T20:43:23.599 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:23.599 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 145 k 2024-03-11T20:43:23.599 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 436 k 2024-03-11T20:43:23.599 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:43:23.604 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9. 31/36 2024-03-11T20:43:23.635 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/36 2024-03-11T20:43:23.660 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/36 2024-03-11T20:43:23.660 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T20:43:23.661 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T20:43:23.661 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T20:43:23.661 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T20:43:23.661 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:23.797 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-immutable-object-cache-19.0.0-1979.g2b0101 734 kB/s | 145 kB 00:00 2024-03-11T20:43:23.797 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:43:23.798 INFO:teuthology.orchestra.run.smithi129.stdout:Total 729 kB/s | 145 kB 00:00 2024-03-11T20:43:23.798 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:43:23.805 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:43:23.805 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:43:23.857 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:43:23.858 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:43:23.992 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:43:24.048 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-11T20:43:24.072 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-11T20:43:24.072 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T20:43:24.072 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T20:43:24.072 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-11T20:43:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:25.064 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:43:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:25.065 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:25.065 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:43:25.094 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/36 2024-03-11T20:43:25.120 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/36 2024-03-11T20:43:25.120 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T20:43:25.121 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T20:43:25.121 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T20:43:25.121 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T20:43:25.121 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:25.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr 2024-03-11T20:43:25.562 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 34/36 2024-03-11T20:43:25.584 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 34/36 2024-03-11T20:43:25.585 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T20:43:25.585 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T20:43:25.585 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T20:43:25.585 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T20:43:25.585 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:25.787 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 35/36 2024-03-11T20:43:25.810 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 35/36 2024-03-11T20:43:25.810 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T20:43:25.810 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T20:43:25.810 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T20:43:25.811 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T20:43:25.811 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:25.853 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:02 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:43:25.877 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 36/36 2024-03-11T20:43:25.949 INFO:teuthology.orchestra.run.smithi129.stdout:Package ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:43:26.000 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:43:26.001 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-11T20:43:26.002 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:43:26.056 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-dashboard 2024-03-11T20:43:26.608 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:03 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:43:26.755 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:43:26.757 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2024-03-11T20:43:26.757 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-11T20:43:26.757 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2024-03-11T20:43:26.757 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:43:26.757 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 1.8 M 2024-03-11T20:43:26.757 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-11T20:43:26.757 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T20:43:26.757 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 34 k 2024-03-11T20:43:26.757 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 16 k 2024-03-11T20:43:26.757 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T20:43:26.757 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T20:43:26.757 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T20:43:26.757 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T20:43:26.758 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T20:43:26.758 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T20:43:26.758 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T20:43:26.758 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T20:43:26.758 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:26.758 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:43:26.758 INFO:teuthology.orchestra.run.smithi129.stdout:====================================================================================== 2024-03-11T20:43:26.758 INFO:teuthology.orchestra.run.smithi129.stdout:Install 12 Packages 2024-03-11T20:43:26.758 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:26.759 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 6.9 M 2024-03-11T20:43:26.759 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 26 M 2024-03-11T20:43:26.759 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:43:27.352 INFO:teuthology.orchestra.run.smithi129.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1979.g2b 85 kB/s | 34 kB 00:00 2024-03-11T20:43:27.485 INFO:teuthology.orchestra.run.smithi129.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1979.g2b0 31 kB/s | 16 kB 00:00 2024-03-11T20:43:27.561 INFO:teuthology.orchestra.run.smithi129.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-11T20:43:27.586 INFO:teuthology.orchestra.run.smithi129.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.3 MB/s | 238 kB 00:00 2024-03-11T20:43:27.671 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 36/36 2024-03-11T20:43:27.671 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/36 2024-03-11T20:43:27.671 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/36 2024-03-11T20:43:27.671 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 3/36 2024-03-11T20:43:27.671 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 4/36 2024-03-11T20:43:27.671 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/36 2024-03-11T20:43:27.671 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9. 6/36 2024-03-11T20:43:27.672 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-11T20:43:27.672 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-11T20:43:27.672 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-11T20:43:27.672 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-11T20:43:27.672 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-11T20:43:27.672 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-11T20:43:27.672 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-11T20:43:27.672 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-11T20:43:27.672 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-11T20:43:27.672 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-11T20:43:27.672 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-11T20:43:27.672 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-11T20:43:27.672 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-11T20:43:27.672 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-11T20:43:27.673 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-11T20:43:27.673 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-11T20:43:27.673 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-11T20:43:27.673 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-11T20:43:27.673 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-11T20:43:27.673 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-11T20:43:27.673 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-11T20:43:27.673 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-11T20:43:27.673 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-11T20:43:27.673 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-11T20:43:27.673 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-11T20:43:27.673 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-11T20:43:27.673 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-11T20:43:27.674 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-11T20:43:27.674 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-11T20:43:27.786 INFO:teuthology.orchestra.run.smithi129.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.7 MB/s | 548 kB 00:00 2024-03-11T20:43:27.837 INFO:teuthology.orchestra.run.smithi129.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 388 kB/s | 19 kB 00:00 2024-03-11T20:43:27.970 INFO:teuthology.orchestra.run.smithi129.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.1 MB/s | 863 kB 00:00 2024-03-11T20:43:28.012 INFO:teuthology.orchestra.run.smithi129.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 12 MB/s | 2.0 MB 00:00 2024-03-11T20:43:28.079 INFO:teuthology.orchestra.run.smithi129.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 462 kB/s | 31 kB 00:00 2024-03-11T20:43:28.130 INFO:teuthology.orchestra.run.smithi129.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-11T20:43:28.155 INFO:teuthology.orchestra.run.smithi129.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 784 kB/s | 144 kB 00:00 2024-03-11T20:43:28.475 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-11T20:43:28.475 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:28.475 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:43:28.475 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:28.475 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:28.475 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:28.475 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:43:28.475 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:28.475 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:28.476 INFO:teuthology.orchestra.run.smithi157.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T20:43:28.476 INFO:teuthology.orchestra.run.smithi157.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T20:43:28.476 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T20:43:28.476 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T20:43:28.476 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T20:43:28.476 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T20:43:28.476 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T20:43:28.476 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T20:43:28.476 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T20:43:28.476 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T20:43:28.476 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T20:43:28.476 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T20:43:28.476 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T20:43:28.477 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T20:43:28.477 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T20:43:28.477 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T20:43:28.477 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T20:43:28.477 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T20:43:28.477 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T20:43:28.477 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T20:43:28.477 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T20:43:28.477 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T20:43:28.477 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T20:43:28.477 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T20:43:28.477 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T20:43:28.477 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T20:43:28.477 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T20:43:28.478 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T20:43:28.478 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T20:43:28.478 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T20:43:28.478 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:28.478 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:43:28.772 INFO:teuthology.orchestra.run.smithi129.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1979.g2b0101 1.0 MB/s | 1.8 MB 00:01 2024-03-11T20:43:28.775 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:43:28.775 INFO:teuthology.orchestra.run.smithi129.stdout:Total 3.4 MB/s | 6.9 MB 00:02 2024-03-11T20:43:28.795 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:43:28.813 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:43:28.814 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:43:28.840 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-base 2024-03-11T20:43:29.000 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:43:29.000 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:43:29.412 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:59 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:43:29.510 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:43:29.513 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:43:29.564 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:43:29.565 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-11T20:43:29.565 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:43:29.581 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-11T20:43:29.616 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-11T20:43:29.650 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install cephadm 2024-03-11T20:43:29.663 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-11T20:43:29.693 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-11T20:43:29.802 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-11T20:43:29.944 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-11T20:43:30.030 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-11T20:43:30.074 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-11T20:43:30.102 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-11T20:43:30.135 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9 10/12 2024-03-11T20:43:30.212 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:00 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:43:30.361 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:43:30.362 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:43:30.362 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-11T20:43:30.362 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:43:30.363 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:43:30.363 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 748 k 2024-03-11T20:43:30.363 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:30.363 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:43:30.363 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:43:30.363 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-11T20:43:30.363 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:30.363 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 748 k 2024-03-11T20:43:30.363 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 755 k 2024-03-11T20:43:30.363 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:43:30.370 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el 11/12 2024-03-11T20:43:30.400 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 12/12 2024-03-11T20:43:30.844 INFO:teuthology.orchestra.run.smithi157.stdout:cephadm-19.0.0-1979.g2b0101eb.el9.noarch.rpm 1.5 MB/s | 748 kB 00:00 2024-03-11T20:43:30.845 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:43:30.845 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.5 MB/s | 748 kB 00:00 2024-03-11T20:43:30.845 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:43:30.850 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:43:30.850 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:43:30.856 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:43:30.856 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:43:30.932 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:43:31.244 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 12/12 2024-03-11T20:43:31.245 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el 1/12 2024-03-11T20:43:31.245 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 2/12 2024-03-11T20:43:31.245 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9 3/12 2024-03-11T20:43:31.245 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-11T20:43:31.245 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-11T20:43:31.245 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-11T20:43:31.245 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-11T20:43:31.245 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-11T20:43:31.245 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-11T20:43:31.245 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-11T20:43:31.245 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-11T20:43:31.376 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-11T20:43:31.449 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-11T20:43:31.601 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-11T20:43:31.601 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:31.601 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:43:31.601 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T20:43:31.601 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:43:31.601 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:43:31.601 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:43:31.602 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T20:43:31.602 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T20:43:31.602 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T20:43:31.602 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T20:43:31.602 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T20:43:31.602 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T20:43:31.602 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T20:43:31.602 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T20:43:31.602 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:31.602 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:43:31.821 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-11T20:43:31.839 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-11T20:43:32.141 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-11T20:43:32.141 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:32.141 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:43:32.142 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:43:32.142 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:32.142 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:43:32.371 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:09 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:43:32.387 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-immutable-object-cache 2024-03-11T20:43:32.522 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:43:32.523 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================================ 2024-03-11T20:43:32.523 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-11T20:43:32.523 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================================ 2024-03-11T20:43:32.523 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:43:32.523 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 7.4 M 2024-03-11T20:43:32.523 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-11T20:43:32.523 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T20:43:32.523 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T20:43:32.524 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T20:43:32.524 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T20:43:32.524 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T20:43:32.524 INFO:teuthology.orchestra.run.smithi129.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T20:43:32.524 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T20:43:32.524 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T20:43:32.524 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T20:43:32.524 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T20:43:32.524 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T20:43:32.524 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:32.524 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:43:32.524 INFO:teuthology.orchestra.run.smithi129.stdout:============================================================================================ 2024-03-11T20:43:32.524 INFO:teuthology.orchestra.run.smithi129.stdout:Install 12 Packages 2024-03-11T20:43:32.524 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:32.525 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 37 M 2024-03-11T20:43:32.525 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 209 M 2024-03-11T20:43:32.525 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:43:32.955 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:02 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:43:32.988 INFO:teuthology.orchestra.run.smithi129.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 220 kB/s | 32 kB 00:00 2024-03-11T20:43:33.022 INFO:teuthology.orchestra.run.smithi129.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 533 kB/s | 17 kB 00:00 2024-03-11T20:43:33.055 INFO:teuthology.orchestra.run.smithi129.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-11T20:43:33.106 INFO:teuthology.orchestra.run.smithi129.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-11T20:43:33.110 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:43:33.111 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:43:33.111 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-11T20:43:33.111 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:43:33.111 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:43:33.111 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 145 k 2024-03-11T20:43:33.111 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:33.111 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:43:33.111 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:43:33.111 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-11T20:43:33.111 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:33.111 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 145 k 2024-03-11T20:43:33.112 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 436 k 2024-03-11T20:43:33.112 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:43:33.264 INFO:teuthology.orchestra.run.smithi129.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-03-11T20:43:33.333 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-immutable-object-cache-19.0.0-1979.g2b0101 655 kB/s | 145 kB 00:00 2024-03-11T20:43:33.334 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:43:33.334 INFO:teuthology.orchestra.run.smithi157.stdout:Total 651 kB/s | 145 kB 00:00 2024-03-11T20:43:33.334 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:43:33.341 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:43:33.341 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:43:33.348 INFO:teuthology.orchestra.run.smithi129.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-11T20:43:33.373 INFO:teuthology.orchestra.run.smithi129.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2024-03-11T20:43:33.393 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:43:33.393 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:43:33.529 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:43:33.592 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-11T20:43:33.616 INFO:teuthology.orchestra.run.smithi129.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 59 MB/s | 16 MB 00:00 2024-03-11T20:43:33.616 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-11T20:43:33.616 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T20:43:33.616 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T20:43:33.616 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:33.749 INFO:teuthology.orchestra.run.smithi129.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-19 8.2 MB/s | 7.4 MB 00:00 2024-03-11T20:43:33.774 INFO:teuthology.orchestra.run.smithi129.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2024-03-11T20:43:33.816 INFO:teuthology.orchestra.run.smithi129.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 976 kB/s | 195 kB 00:00 2024-03-11T20:43:33.900 INFO:teuthology.orchestra.run.smithi129.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-11T20:43:33.902 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:43:33.902 INFO:teuthology.orchestra.run.smithi129.stdout:Total 27 MB/s | 37 MB 00:01 2024-03-11T20:43:33.939 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:43:33.962 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:43:33.963 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:43:34.319 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:43:34.319 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:43:34.648 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-11T20:43:34.648 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:34.648 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:43:34.648 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:43:34.648 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:34.648 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:43:34.923 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr 2024-03-11T20:43:35.022 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:43:35.089 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-11T20:43:35.121 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-11T20:43:35.216 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-11T20:43:35.257 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-11T20:43:35.437 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T20:43:35.465 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T20:43:35.498 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:05 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:43:35.541 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-11T20:43:35.596 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:43:35.648 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:43:35.649 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-11T20:43:35.649 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:43:35.714 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-dashboard 2024-03-11T20:43:36.184 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-11T20:43:36.278 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:06 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:43:36.294 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-11T20:43:36.428 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:43:36.429 INFO:teuthology.orchestra.run.smithi157.stdout:====================================================================================== 2024-03-11T20:43:36.429 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-11T20:43:36.429 INFO:teuthology.orchestra.run.smithi157.stdout:====================================================================================== 2024-03-11T20:43:36.430 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:43:36.430 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 1.8 M 2024-03-11T20:43:36.430 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-11T20:43:36.430 INFO:teuthology.orchestra.run.smithi157.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T20:43:36.430 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 34 k 2024-03-11T20:43:36.430 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 16 k 2024-03-11T20:43:36.430 INFO:teuthology.orchestra.run.smithi157.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T20:43:36.430 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T20:43:36.430 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T20:43:36.430 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T20:43:36.430 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T20:43:36.430 INFO:teuthology.orchestra.run.smithi157.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T20:43:36.430 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T20:43:36.431 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T20:43:36.431 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:36.431 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:43:36.431 INFO:teuthology.orchestra.run.smithi157.stdout:====================================================================================== 2024-03-11T20:43:36.431 INFO:teuthology.orchestra.run.smithi157.stdout:Install 12 Packages 2024-03-11T20:43:36.431 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:36.431 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 6.9 M 2024-03-11T20:43:36.432 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 26 M 2024-03-11T20:43:36.432 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:43:36.953 INFO:teuthology.orchestra.run.smithi157.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1979.g2b 183 kB/s | 34 kB 00:00 2024-03-11T20:43:37.054 INFO:teuthology.orchestra.run.smithi157.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1979.g2b0 58 kB/s | 16 kB 00:00 2024-03-11T20:43:37.237 INFO:teuthology.orchestra.run.smithi157.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-03-11T20:43:37.262 INFO:teuthology.orchestra.run.smithi157.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 238 kB 00:00 2024-03-11T20:43:37.338 INFO:teuthology.orchestra.run.smithi157.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1979.g2b0101e 3.1 MB/s | 1.8 MB 00:00 2024-03-11T20:43:37.446 INFO:teuthology.orchestra.run.smithi157.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.0 MB/s | 863 kB 00:00 2024-03-11T20:43:37.518 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-11T20:43:37.646 INFO:teuthology.orchestra.run.smithi157.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 63 kB/s | 19 kB 00:00 2024-03-11T20:43:37.947 INFO:teuthology.orchestra.run.smithi157.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 801 kB/s | 548 kB 00:00 2024-03-11T20:43:38.005 INFO:teuthology.orchestra.run.smithi157.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 403 kB/s | 144 kB 00:00 2024-03-11T20:43:38.055 INFO:teuthology.orchestra.run.smithi157.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 283 kB/s | 31 kB 00:00 2024-03-11T20:43:38.197 INFO:teuthology.orchestra.run.smithi157.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 983 kB/s | 188 kB 00:00 2024-03-11T20:43:38.289 INFO:teuthology.orchestra.run.smithi157.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.4 MB/s | 2.0 MB 00:00 2024-03-11T20:43:38.292 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:43:38.292 INFO:teuthology.orchestra.run.smithi157.stdout:Total 3.7 MB/s | 6.9 MB 00:01 2024-03-11T20:43:38.311 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:43:38.328 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:43:38.328 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:43:38.517 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:43:38.518 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:43:38.571 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-11T20:43:38.608 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 12/12 2024-03-11T20:43:39.036 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:43:39.123 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-11T20:43:39.158 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-11T20:43:39.196 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-11T20:43:39.227 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-11T20:43:39.331 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-11T20:43:39.478 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-11T20:43:39.563 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-11T20:43:39.614 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-11T20:43:39.652 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-11T20:43:39.684 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9 10/12 2024-03-11T20:43:39.930 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el 11/12 2024-03-11T20:43:39.958 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 12/12 2024-03-11T20:43:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 12/12 2024-03-11T20:43:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el 1/12 2024-03-11T20:43:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 2/12 2024-03-11T20:43:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9 3/12 2024-03-11T20:43:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-11T20:43:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-11T20:43:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-11T20:43:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-11T20:43:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-11T20:43:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-11T20:43:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-11T20:43:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-11T20:43:41.181 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-11T20:43:41.181 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:41.181 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:43:41.181 INFO:teuthology.orchestra.run.smithi157.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T20:43:41.181 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:43:41.181 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:43:41.181 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:43:41.181 INFO:teuthology.orchestra.run.smithi157.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T20:43:41.181 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T20:43:41.182 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T20:43:41.182 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T20:43:41.182 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T20:43:41.182 INFO:teuthology.orchestra.run.smithi157.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T20:43:41.182 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T20:43:41.182 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T20:43:41.182 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:41.182 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:43:41.193 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 12/12 2024-03-11T20:43:41.193 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 1/12 2024-03-11T20:43:41.193 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-11T20:43:41.193 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-11T20:43:41.193 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-11T20:43:41.193 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T20:43:41.193 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T20:43:41.193 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-11T20:43:41.193 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-11T20:43:41.193 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-11T20:43:41.193 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-11T20:43:41.193 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-11T20:43:41.495 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-11T20:43:41.626 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-11T20:43:41.626 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:41.626 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:43:41.627 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:43:41.627 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T20:43:41.627 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T20:43:41.627 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T20:43:41.627 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T20:43:41.627 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T20:43:41.627 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T20:43:41.627 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T20:43:41.627 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T20:43:41.627 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T20:43:41.627 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T20:43:41.627 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T20:43:41.627 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:41.627 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:43:41.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-rook 2024-03-11T20:43:42.052 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:12 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:43:42.204 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:43:42.205 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================================ 2024-03-11T20:43:42.205 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-11T20:43:42.205 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================================ 2024-03-11T20:43:42.206 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:43:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 7.4 M 2024-03-11T20:43:42.206 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-11T20:43:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T20:43:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T20:43:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T20:43:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T20:43:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T20:43:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T20:43:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T20:43:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T20:43:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T20:43:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T20:43:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T20:43:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:42.207 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:43:42.207 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================================ 2024-03-11T20:43:42.207 INFO:teuthology.orchestra.run.smithi157.stdout:Install 12 Packages 2024-03-11T20:43:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:42.208 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 37 M 2024-03-11T20:43:42.208 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 209 M 2024-03-11T20:43:42.208 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:43:42.551 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:19 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:43:42.705 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:43:42.706 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-03-11T20:43:42.706 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-11T20:43:42.706 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-03-11T20:43:42.706 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:43:42.706 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 49 k 2024-03-11T20:43:42.706 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-11T20:43:42.707 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T20:43:42.707 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T20:43:42.707 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T20:43:42.707 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T20:43:42.707 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T20:43:42.707 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T20:43:42.707 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T20:43:42.707 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T20:43:42.707 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T20:43:42.707 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T20:43:42.707 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T20:43:42.707 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T20:43:42.707 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:42.707 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:43:42.708 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-03-11T20:43:42.708 INFO:teuthology.orchestra.run.smithi129.stdout:Install 13 Packages 2024-03-11T20:43:42.708 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:42.708 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 2.0 M 2024-03-11T20:43:42.708 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 26 M 2024-03-11T20:43:42.708 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:43:42.887 INFO:teuthology.orchestra.run.smithi157.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 232 kB/s | 32 kB 00:00 2024-03-11T20:43:42.894 INFO:teuthology.orchestra.run.smithi129.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 129 kB/s | 18 kB 00:00 2024-03-11T20:43:42.920 INFO:teuthology.orchestra.run.smithi129.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 148 kB/s | 24 kB 00:00 2024-03-11T20:43:42.929 INFO:teuthology.orchestra.run.smithi157.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 415 kB/s | 17 kB 00:00 2024-03-11T20:43:42.971 INFO:teuthology.orchestra.run.smithi157.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 845 kB/s | 34 kB 00:00 2024-03-11T20:43:42.978 INFO:teuthology.orchestra.run.smithi129.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-11T20:43:43.003 INFO:teuthology.orchestra.run.smithi129.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-11T20:43:43.021 INFO:teuthology.orchestra.run.smithi157.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-11T20:43:43.029 INFO:teuthology.orchestra.run.smithi129.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-11T20:43:43.046 INFO:teuthology.orchestra.run.smithi129.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-11T20:43:43.163 INFO:teuthology.orchestra.run.smithi129.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-11T20:43:43.188 INFO:teuthology.orchestra.run.smithi129.stdout:(8/13): ceph-mgr-rook-19.0.0-1979.g2b0101eb.el9 114 kB/s | 49 kB 00:00 2024-03-11T20:43:43.196 INFO:teuthology.orchestra.run.smithi157.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-03-11T20:43:43.255 INFO:teuthology.orchestra.run.smithi157.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-11T20:43:43.280 INFO:teuthology.orchestra.run.smithi157.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-11T20:43:43.338 INFO:teuthology.orchestra.run.smithi129.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 104 kB/s | 32 kB 00:00 2024-03-11T20:43:43.506 INFO:teuthology.orchestra.run.smithi157.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 63 MB/s | 16 MB 00:00 2024-03-11T20:43:43.748 INFO:teuthology.orchestra.run.smithi157.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 896 kB/s | 195 kB 00:00 2024-03-11T20:43:43.789 INFO:teuthology.orchestra.run.smithi157.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-11T20:43:43.914 INFO:teuthology.orchestra.run.smithi129.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 103 kB/s | 59 kB 00:00 2024-03-11T20:43:43.956 INFO:teuthology.orchestra.run.smithi157.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-11T20:43:44.273 INFO:teuthology.orchestra.run.smithi157.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.9 MB/s | 7.4 MB 00:01 2024-03-11T20:43:44.276 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:43:44.277 INFO:teuthology.orchestra.run.smithi157.stdout:Total 18 MB/s | 37 MB 00:02 2024-03-11T20:43:44.317 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:43:44.341 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:43:44.341 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:43:44.706 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:43:44.707 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:43:44.764 INFO:teuthology.orchestra.run.smithi129.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 105 kB/s | 90 kB 00:00 2024-03-11T20:43:45.006 INFO:teuthology.orchestra.run.smithi129.stdout:(12/13): python3-google-auth-2.28.0-1.el9.noarc 110 kB/s | 203 kB 00:01 2024-03-11T20:43:45.417 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:43:45.490 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-11T20:43:45.519 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-11T20:43:45.613 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-11T20:43:45.655 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-11T20:43:45.825 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T20:43:45.854 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T20:43:45.931 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-11T20:43:46.607 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-11T20:43:46.725 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-11T20:43:47.978 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-11T20:43:49.005 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-11T20:43:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 12/12 2024-03-11T20:43:51.717 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 12/12 2024-03-11T20:43:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 1/12 2024-03-11T20:43:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-11T20:43:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-11T20:43:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-11T20:43:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T20:43:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T20:43:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-11T20:43:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-11T20:43:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-11T20:43:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-11T20:43:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-11T20:43:52.189 INFO:teuthology.orchestra.run.smithi129.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 116 kB/s | 1.0 MB 00:09 2024-03-11T20:43:52.191 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:43:52.192 INFO:teuthology.orchestra.run.smithi129.stdout:Total 221 kB/s | 2.0 MB 00:09 2024-03-11T20:43:52.192 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:43:52.207 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:43:52.207 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:43:52.213 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-11T20:43:52.213 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:52.213 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:43:52.213 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:43:52.213 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T20:43:52.213 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T20:43:52.213 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T20:43:52.214 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T20:43:52.214 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T20:43:52.214 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T20:43:52.214 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T20:43:52.214 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T20:43:52.214 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T20:43:52.214 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T20:43:52.214 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T20:43:52.214 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:52.214 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:43:52.314 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:43:52.314 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:43:52.591 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-rook 2024-03-11T20:43:52.861 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:43:53.025 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-11T20:43:53.082 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-11T20:43:53.137 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-11T20:43:53.160 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:23 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:43:53.175 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-11T20:43:53.216 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-11T20:43:53.309 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-11T20:43:53.312 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:43:53.313 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-11T20:43:53.313 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-11T20:43:53.313 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-11T20:43:53.313 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:43:53.313 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 49 k 2024-03-11T20:43:53.314 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-11T20:43:53.314 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T20:43:53.314 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T20:43:53.314 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T20:43:53.314 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T20:43:53.314 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T20:43:53.314 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T20:43:53.314 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T20:43:53.314 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T20:43:53.314 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T20:43:53.314 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T20:43:53.314 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T20:43:53.314 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T20:43:53.314 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:53.315 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:43:53.315 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-11T20:43:53.315 INFO:teuthology.orchestra.run.smithi157.stdout:Install 13 Packages 2024-03-11T20:43:53.315 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:53.315 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 2.0 M 2024-03-11T20:43:53.315 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 26 M 2024-03-11T20:43:53.316 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:43:53.435 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-11T20:43:53.501 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-11T20:43:53.788 INFO:teuthology.orchestra.run.smithi157.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 151 kB/s | 18 kB 00:00 2024-03-11T20:43:53.813 INFO:teuthology.orchestra.run.smithi157.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 169 kB/s | 24 kB 00:00 2024-03-11T20:43:53.872 INFO:teuthology.orchestra.run.smithi157.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-11T20:43:53.897 INFO:teuthology.orchestra.run.smithi157.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-11T20:43:53.922 INFO:teuthology.orchestra.run.smithi157.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-11T20:43:53.947 INFO:teuthology.orchestra.run.smithi157.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-11T20:43:54.048 INFO:teuthology.orchestra.run.smithi157.stdout:(7/13): ceph-mgr-rook-19.0.0-1979.g2b0101eb.el9 130 kB/s | 49 kB 00:00 2024-03-11T20:43:54.073 INFO:teuthology.orchestra.run.smithi157.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 120 kB/s | 15 kB 00:00 2024-03-11T20:43:54.098 INFO:teuthology.orchestra.run.smithi157.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 183 kB/s | 32 kB 00:00 2024-03-11T20:43:54.214 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-11T20:43:54.257 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-11T20:43:54.290 INFO:teuthology.orchestra.run.smithi157.stdout:(10/13): python3-google-auth-2.28.0-1.el9.noarc 837 kB/s | 203 kB 00:00 2024-03-11T20:43:54.291 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-11T20:43:54.315 INFO:teuthology.orchestra.run.smithi157.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 272 kB/s | 59 kB 00:00 2024-03-11T20:43:54.339 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-11T20:43:54.357 INFO:teuthology.orchestra.run.smithi157.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-11T20:43:54.361 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 13/13 2024-03-11T20:43:54.382 INFO:teuthology.orchestra.run.smithi157.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.3 MB/s | 1.0 MB 00:00 2024-03-11T20:43:54.384 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:43:54.384 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-11T20:43:54.385 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:43:54.398 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:43:54.398 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:43:54.498 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:43:54.498 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:43:55.027 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:43:55.184 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 13/13 2024-03-11T20:43:55.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 1/13 2024-03-11T20:43:55.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-11T20:43:55.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-11T20:43:55.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-11T20:43:55.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-11T20:43:55.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-11T20:43:55.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-11T20:43:55.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-11T20:43:55.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-11T20:43:55.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-11T20:43:55.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-11T20:43:55.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-11T20:43:55.189 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-11T20:43:55.250 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-11T20:43:55.306 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-11T20:43:55.343 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-11T20:43:55.384 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-11T20:43:55.477 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-11T20:43:55.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-11T20:43:55.549 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:55.549 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:43:55.549 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:43:55.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T20:43:55.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T20:43:55.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T20:43:55.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T20:43:55.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T20:43:55.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T20:43:55.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T20:43:55.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T20:43:55.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T20:43:55.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T20:43:55.549 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T20:43:55.550 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T20:43:55.550 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:55.550 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:43:55.579 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-11T20:43:55.636 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-11T20:43:55.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-cephadm 2024-03-11T20:43:56.361 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:33 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:43:56.367 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-11T20:43:56.408 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-11T20:43:56.442 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-11T20:43:56.490 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-11T20:43:56.511 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 13/13 2024-03-11T20:43:56.512 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:43:56.513 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-03-11T20:43:56.513 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-11T20:43:56.513 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-03-11T20:43:56.513 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:43:56.513 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 139 k 2024-03-11T20:43:56.513 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-11T20:43:56.513 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T20:43:56.513 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T20:43:56.513 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T20:43:56.513 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T20:43:56.513 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T20:43:56.514 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:56.514 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:43:56.514 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================== 2024-03-11T20:43:56.514 INFO:teuthology.orchestra.run.smithi129.stdout:Install 6 Packages 2024-03-11T20:43:56.514 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:43:56.514 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 6.8 M 2024-03-11T20:43:56.515 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 33 M 2024-03-11T20:43:56.515 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:43:57.403 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 13/13 2024-03-11T20:43:57.403 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 1/13 2024-03-11T20:43:57.403 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-11T20:43:57.404 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-11T20:43:57.404 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-11T20:43:57.404 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-11T20:43:57.404 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-11T20:43:57.404 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-11T20:43:57.404 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-11T20:43:57.404 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-11T20:43:57.404 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-11T20:43:57.404 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-11T20:43:57.404 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-11T20:43:57.426 INFO:teuthology.orchestra.run.smithi129.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-11T20:43:57.451 INFO:teuthology.orchestra.run.smithi129.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1979.g2b0101eb.e 464 kB/s | 139 kB 00:00 2024-03-11T20:43:57.476 INFO:teuthology.orchestra.run.smithi129.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 766 kB/s | 248 kB 00:00 2024-03-11T20:43:57.660 INFO:teuthology.orchestra.run.smithi129.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 315 kB/s | 58 kB 00:00 2024-03-11T20:43:57.676 INFO:teuthology.orchestra.run.smithi129.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 255 kB/s | 51 kB 00:00 2024-03-11T20:43:57.778 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-11T20:43:57.778 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:57.778 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:43:57.778 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:43:57.778 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T20:43:57.778 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T20:43:57.778 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T20:43:57.778 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T20:43:57.778 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T20:43:57.779 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T20:43:57.779 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T20:43:57.779 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T20:43:57.779 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T20:43:57.779 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T20:43:57.779 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T20:43:57.779 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T20:43:57.779 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:57.779 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:43:57.885 INFO:teuthology.orchestra.run.smithi129.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-11T20:43:57.888 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:43:57.888 INFO:teuthology.orchestra.run.smithi129.stdout:Total 4.9 MB/s | 6.8 MB 00:01 2024-03-11T20:43:57.901 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:43:57.911 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:43:57.911 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:43:58.005 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:43:58.005 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:43:58.038 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-cephadm 2024-03-11T20:43:58.303 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:43:58.474 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-11T20:43:58.529 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-11T20:43:58.604 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:28 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:43:58.756 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:43:58.757 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-11T20:43:58.757 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-11T20:43:58.757 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-11T20:43:58.757 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:43:58.757 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 139 k 2024-03-11T20:43:58.757 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-11T20:43:58.757 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T20:43:58.758 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T20:43:58.758 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T20:43:58.758 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T20:43:58.758 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T20:43:58.758 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:58.758 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:43:58.758 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================== 2024-03-11T20:43:58.758 INFO:teuthology.orchestra.run.smithi157.stdout:Install 6 Packages 2024-03-11T20:43:58.759 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:43:58.760 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 6.8 M 2024-03-11T20:43:58.760 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 33 M 2024-03-11T20:43:58.760 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:43:59.049 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-11T20:43:59.123 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-11T20:43:59.184 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-11T20:43:59.208 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 6/6 2024-03-11T20:43:59.417 INFO:teuthology.orchestra.run.smithi157.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-11T20:43:59.445 INFO:teuthology.orchestra.run.smithi157.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1979.g2b0101eb.e 455 kB/s | 139 kB 00:00 2024-03-11T20:43:59.720 INFO:teuthology.orchestra.run.smithi157.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 427 kB/s | 248 kB 00:00 2024-03-11T20:43:59.745 INFO:teuthology.orchestra.run.smithi157.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 192 kB/s | 58 kB 00:00 2024-03-11T20:43:59.963 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 6/6 2024-03-11T20:43:59.963 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/6 2024-03-11T20:43:59.963 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-11T20:43:59.964 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-11T20:43:59.964 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-11T20:43:59.964 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-11T20:43:59.987 INFO:teuthology.orchestra.run.smithi157.stdout:(5/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 962 kB/s | 548 kB 00:00 2024-03-11T20:44:00.029 INFO:teuthology.orchestra.run.smithi157.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 165 kB/s | 51 kB 00:00 2024-03-11T20:44:00.033 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:44:00.033 INFO:teuthology.orchestra.run.smithi157.stdout:Total 5.3 MB/s | 6.8 MB 00:01 2024-03-11T20:44:00.045 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:44:00.056 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:44:00.056 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:44:00.149 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:44:00.151 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:44:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-11T20:44:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:00.274 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:44:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:44:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T20:44:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T20:44:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T20:44:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T20:44:00.275 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T20:44:00.275 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:00.275 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:44:00.448 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:44:00.500 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-fuse 2024-03-11T20:44:00.652 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-11T20:44:00.706 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-11T20:44:01.062 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:38 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:44:01.210 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:44:01.211 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-11T20:44:01.211 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:01.211 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:44:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 888 k 2024-03-11T20:44:01.211 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-11T20:44:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T20:44:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:01.212 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:44:01.212 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:01.212 INFO:teuthology.orchestra.run.smithi129.stdout:Install 2 Packages 2024-03-11T20:44:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:01.212 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 966 k 2024-03-11T20:44:01.212 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 2.8 M 2024-03-11T20:44:01.213 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:44:01.226 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-11T20:44:01.316 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-11T20:44:01.346 INFO:teuthology.orchestra.run.smithi129.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 590 kB/s | 79 kB 00:00 2024-03-11T20:44:01.386 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-11T20:44:01.409 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 6/6 2024-03-11T20:44:02.013 INFO:teuthology.orchestra.run.smithi129.stdout:(2/2): ceph-fuse-19.0.0-1979.g2b0101eb.el9.x86_ 1.1 MB/s | 888 kB 00:00 2024-03-11T20:44:02.014 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:44:02.014 INFO:teuthology.orchestra.run.smithi129.stdout:Total 1.2 MB/s | 966 kB 00:00 2024-03-11T20:44:02.015 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:44:02.035 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:44:02.036 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:44:02.113 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:44:02.113 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:44:02.231 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 6/6 2024-03-11T20:44:02.231 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/6 2024-03-11T20:44:02.231 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-11T20:44:02.231 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-11T20:44:02.232 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-11T20:44:02.232 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-11T20:44:02.299 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:44:02.443 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-11T20:44:02.504 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/2 2024-03-11T20:44:02.532 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-11T20:44:02.532 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:02.532 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:44:02.532 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:44:02.532 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T20:44:02.532 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T20:44:02.532 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T20:44:02.532 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T20:44:02.532 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T20:44:02.532 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:02.532 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:44:02.810 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-fuse 2024-03-11T20:44:03.179 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/2 2024-03-11T20:44:03.180 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/2 2024-03-11T20:44:03.376 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:33 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:44:03.480 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-11T20:44:03.480 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:03.480 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:44:03.480 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-11T20:44:03.480 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:03.480 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:44:03.527 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:44:03.528 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:03.528 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-11T20:44:03.528 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:03.528 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:44:03.528 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 888 k 2024-03-11T20:44:03.529 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-11T20:44:03.529 INFO:teuthology.orchestra.run.smithi157.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T20:44:03.529 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:03.529 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:44:03.529 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:03.529 INFO:teuthology.orchestra.run.smithi157.stdout:Install 2 Packages 2024-03-11T20:44:03.529 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:03.529 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 966 k 2024-03-11T20:44:03.529 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 2.8 M 2024-03-11T20:44:03.529 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:44:03.655 INFO:teuthology.orchestra.run.smithi157.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 624 kB/s | 79 kB 00:00 2024-03-11T20:44:03.753 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-volume 2024-03-11T20:44:04.164 INFO:teuthology.orchestra.run.smithi157.stdout:(2/2): ceph-fuse-19.0.0-1979.g2b0101eb.el9.x86_ 1.4 MB/s | 888 kB 00:00 2024-03-11T20:44:04.165 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:44:04.165 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.5 MB/s | 966 kB 00:00 2024-03-11T20:44:04.166 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:44:04.187 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:44:04.187 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:44:04.264 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:44:04.265 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:44:04.325 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:41 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:44:04.454 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:44:04.475 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:44:04.476 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:04.476 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-03-11T20:44:04.476 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:04.476 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:44:04.476 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 264 k 2024-03-11T20:44:04.476 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-03-11T20:44:04.476 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T20:44:04.476 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:04.477 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:44:04.477 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:04.477 INFO:teuthology.orchestra.run.smithi129.stdout:Install 2 Packages 2024-03-11T20:44:04.477 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:04.477 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 332 k 2024-03-11T20:44:04.477 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 1.5 M 2024-03-11T20:44:04.478 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:44:04.597 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-11T20:44:04.619 INFO:teuthology.orchestra.run.smithi129.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 486 kB/s | 69 kB 00:00 2024-03-11T20:44:04.664 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/2 2024-03-11T20:44:04.786 INFO:teuthology.orchestra.run.smithi129.stdout:(2/2): ceph-volume-19.0.0-1979.g2b0101eb.el9.no 856 kB/s | 264 kB 00:00 2024-03-11T20:44:04.787 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:44:04.788 INFO:teuthology.orchestra.run.smithi129.stdout:Total 1.0 MB/s | 332 kB 00:00 2024-03-11T20:44:04.788 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:44:04.796 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:44:04.796 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:44:04.836 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:44:04.836 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:44:04.986 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:44:05.170 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-11T20:44:05.216 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2/2 2024-03-11T20:44:05.231 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2/2 2024-03-11T20:44:05.232 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T20:44:05.232 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T20:44:05.232 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:05.364 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/2 2024-03-11T20:44:05.364 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/2 2024-03-11T20:44:05.680 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-11T20:44:05.680 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:05.680 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:44:05.680 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-11T20:44:05.681 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:05.681 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:44:05.974 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 1/2 2024-03-11T20:44:05.994 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-volume 2024-03-11T20:44:06.329 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-11T20:44:06.330 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:06.330 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:44:06.330 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:44:06.330 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T20:44:06.330 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:06.330 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:44:06.572 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:36 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:44:06.627 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librados-devel 2024-03-11T20:44:06.725 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:44:06.726 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:06.726 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-11T20:44:06.726 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:06.726 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:44:06.726 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 264 k 2024-03-11T20:44:06.726 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-11T20:44:06.726 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T20:44:06.726 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:06.726 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:44:06.726 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:06.726 INFO:teuthology.orchestra.run.smithi157.stdout:Install 2 Packages 2024-03-11T20:44:06.726 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:06.727 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 332 k 2024-03-11T20:44:06.727 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 1.5 M 2024-03-11T20:44:06.727 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:44:06.860 INFO:teuthology.orchestra.run.smithi157.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 516 kB/s | 69 kB 00:00 2024-03-11T20:44:06.985 INFO:teuthology.orchestra.run.smithi157.stdout:(2/2): ceph-volume-19.0.0-1979.g2b0101eb.el9.no 1.0 MB/s | 264 kB 00:00 2024-03-11T20:44:06.987 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:44:06.987 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-11T20:44:06.987 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:44:06.998 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:44:06.998 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:44:07.040 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:44:07.040 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:44:07.188 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:44:07.198 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:44 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:44:07.350 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:44:07.351 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-03-11T20:44:07.351 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:07.351 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:44:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 127 k 2024-03-11T20:44:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:07.351 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:44:07.351 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:07.351 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-03-11T20:44:07.351 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:07.352 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 127 k 2024-03-11T20:44:07.352 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 456 k 2024-03-11T20:44:07.352 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:44:07.369 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-11T20:44:07.413 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2/2 2024-03-11T20:44:07.429 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2/2 2024-03-11T20:44:07.429 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T20:44:07.429 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T20:44:07.429 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:07.561 INFO:teuthology.orchestra.run.smithi129.stdout:librados-devel-19.0.0-1979.g2b0101eb.el9.x86_64 604 kB/s | 127 kB 00:00 2024-03-11T20:44:07.562 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:44:07.562 INFO:teuthology.orchestra.run.smithi129.stdout:Total 600 kB/s | 127 kB 00:00 2024-03-11T20:44:07.562 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:44:07.568 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:44:07.568 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:44:07.623 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:44:07.623 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:44:07.805 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:44:07.935 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:08.168 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 1/2 2024-03-11T20:44:08.409 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:08.540 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-11T20:44:08.540 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:08.540 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:44:08.540 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T20:44:08.540 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T20:44:08.540 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:08.540 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:44:08.750 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:08.750 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:08.750 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:44:08.750 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:44:08.750 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:08.750 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:44:08.861 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados-devel 2024-03-11T20:44:08.979 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install libcephfs2 2024-03-11T20:44:09.438 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:39 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:44:09.566 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:46 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:44:09.590 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:44:09.591 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:09.591 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-11T20:44:09.591 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:09.591 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:44:09.591 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 127 k 2024-03-11T20:44:09.591 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:09.591 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:44:09.591 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:09.591 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-11T20:44:09.591 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:09.591 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 127 k 2024-03-11T20:44:09.591 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 456 k 2024-03-11T20:44:09.591 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:44:09.669 INFO:teuthology.orchestra.run.smithi129.stdout:Package libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:44:09.721 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:44:09.722 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-11T20:44:09.722 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:44:09.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install libcephfs-devel 2024-03-11T20:44:10.100 INFO:teuthology.orchestra.run.smithi157.stdout:librados-devel-19.0.0-1979.g2b0101eb.el9.x86_64 249 kB/s | 127 kB 00:00 2024-03-11T20:44:10.100 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:44:10.101 INFO:teuthology.orchestra.run.smithi157.stdout:Total 248 kB/s | 127 kB 00:00 2024-03-11T20:44:10.101 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:44:10.106 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:44:10.106 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:44:10.164 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:44:10.165 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:44:10.351 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:47 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:44:10.365 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:44:10.474 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:10.502 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:44:10.503 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:10.503 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-03-11T20:44:10.503 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:10.503 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:44:10.503 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 31 k 2024-03-11T20:44:10.503 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:10.503 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:44:10.503 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:10.503 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-03-11T20:44:10.504 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:10.504 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 31 k 2024-03-11T20:44:10.504 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 140 k 2024-03-11T20:44:10.504 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:44:10.981 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:11.306 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:11.306 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:11.306 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:44:11.306 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:44:11.306 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:11.306 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:44:11.392 INFO:teuthology.orchestra.run.smithi129.stdout:libcephfs-devel-19.0.0-1979.g2b0101eb.el9.x86_6 35 kB/s | 31 kB 00:00 2024-03-11T20:44:11.392 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:44:11.393 INFO:teuthology.orchestra.run.smithi129.stdout:Total 35 kB/s | 31 kB 00:00 2024-03-11T20:44:11.393 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:44:11.396 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:44:11.396 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:44:11.420 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:44:11.420 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:44:11.524 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:44:11.579 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs2 2024-03-11T20:44:11.633 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:12.016 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:12.155 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:42 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:44:12.254 INFO:teuthology.orchestra.run.smithi157.stdout:Package libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:44:12.280 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:12.280 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:12.280 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:44:12.280 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:44:12.280 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:12.280 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:44:12.307 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:44:12.308 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-11T20:44:12.308 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:44:12.371 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs-devel 2024-03-11T20:44:12.455 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librados2 2024-03-11T20:44:12.936 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:42 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:44:13.012 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:50 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:44:13.086 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:44:13.087 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:13.087 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-11T20:44:13.087 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:13.087 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:44:13.087 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 31 k 2024-03-11T20:44:13.087 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:13.087 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:44:13.087 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:13.087 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-11T20:44:13.088 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:13.088 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 31 k 2024-03-11T20:44:13.088 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 140 k 2024-03-11T20:44:13.088 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:44:13.107 INFO:teuthology.orchestra.run.smithi129.stdout:Package librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:44:13.160 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:44:13.161 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-11T20:44:13.161 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:44:13.218 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librbd1 2024-03-11T20:44:13.261 INFO:teuthology.orchestra.run.smithi157.stdout:libcephfs-devel-19.0.0-1979.g2b0101eb.el9.x86_6 180 kB/s | 31 kB 00:00 2024-03-11T20:44:13.262 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:44:13.262 INFO:teuthology.orchestra.run.smithi157.stdout:Total 178 kB/s | 31 kB 00:00 2024-03-11T20:44:13.262 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:44:13.265 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:44:13.266 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:44:13.295 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:44:13.295 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:44:13.392 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:44:13.509 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:13.775 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:50 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:44:13.873 INFO:teuthology.orchestra.run.smithi129.stdout:Package librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:44:13.901 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:13.924 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:44:13.925 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-11T20:44:13.925 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:44:13.984 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rados 2024-03-11T20:44:14.184 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:14.184 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:14.184 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:44:14.184 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:44:14.184 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:14.184 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:44:14.380 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados2 2024-03-11T20:44:14.549 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:51 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:44:14.646 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:44:14.700 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:44:14.701 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-11T20:44:14.701 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:44:14.778 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rgw 2024-03-11T20:44:14.938 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:44 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:44:15.035 INFO:teuthology.orchestra.run.smithi157.stdout:Package librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:44:15.086 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:44:15.087 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-11T20:44:15.087 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:44:15.146 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librbd1 2024-03-11T20:44:15.340 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:52 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:44:15.436 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:44:15.487 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:44:15.488 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-11T20:44:15.488 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:44:15.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-cephfs 2024-03-11T20:44:15.711 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:45 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:44:15.808 INFO:teuthology.orchestra.run.smithi157.stdout:Package librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:44:15.859 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:44:15.860 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-11T20:44:15.860 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:44:15.921 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rados 2024-03-11T20:44:16.107 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:53 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:44:16.205 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:44:16.257 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:44:16.257 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-11T20:44:16.258 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:44:16.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rbd 2024-03-11T20:44:16.491 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:46 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:44:16.588 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:44:16.640 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:44:16.641 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-11T20:44:16.641 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:44:16.713 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rgw 2024-03-11T20:44:16.955 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:53 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:44:17.051 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:44:17.102 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:44:17.103 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-03-11T20:44:17.103 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:44:17.181 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-fuse 2024-03-11T20:44:17.279 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:47 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:44:17.375 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:44:17.427 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:44:17.428 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-11T20:44:17.429 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:44:17.492 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-cephfs 2024-03-11T20:44:17.742 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:54 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:44:17.891 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:44:17.892 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:17.892 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-03-11T20:44:17.892 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:17.892 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:44:17.892 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 85 k 2024-03-11T20:44:17.892 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:17.892 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:44:17.892 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:17.892 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-03-11T20:44:17.892 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:17.892 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 85 k 2024-03-11T20:44:17.892 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 231 k 2024-03-11T20:44:17.893 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:44:18.054 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:48 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:44:18.151 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:44:18.203 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:44:18.204 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-11T20:44:18.204 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:44:18.245 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-fuse-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 242 kB/s | 85 kB 00:00 2024-03-11T20:44:18.246 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:44:18.246 INFO:teuthology.orchestra.run.smithi129.stdout:Total 241 kB/s | 85 kB 00:00 2024-03-11T20:44:18.246 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:44:18.253 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:44:18.253 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:44:18.261 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rbd 2024-03-11T20:44:18.308 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:44:18.308 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:44:18.441 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:44:18.535 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:18.827 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:48 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:44:18.925 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T20:44:18.929 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:18.977 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:44:18.978 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-03-11T20:44:18.978 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:44:19.045 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-fuse 2024-03-11T20:44:19.271 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:19.272 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:19.272 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:44:19.272 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:44:19.272 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:19.272 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:44:19.513 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-mirror 2024-03-11T20:44:19.608 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:49 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:44:19.755 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:44:19.756 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:19.756 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-03-11T20:44:19.756 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:19.757 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:44:19.757 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 85 k 2024-03-11T20:44:19.757 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:19.757 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:44:19.757 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:19.757 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-11T20:44:19.757 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:19.757 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 85 k 2024-03-11T20:44:19.757 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 231 k 2024-03-11T20:44:19.757 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:44:19.956 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-fuse-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 428 kB/s | 85 kB 00:00 2024-03-11T20:44:19.957 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:44:19.957 INFO:teuthology.orchestra.run.smithi157.stdout:Total 426 kB/s | 85 kB 00:00 2024-03-11T20:44:19.957 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:44:19.964 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:44:19.964 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:44:20.020 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:44:20.020 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:44:20.100 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:57 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:44:20.153 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:44:20.250 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:44:20.250 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:20.251 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-03-11T20:44:20.251 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:20.251 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:44:20.251 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.1 M 2024-03-11T20:44:20.251 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:20.251 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:44:20.251 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:20.251 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-03-11T20:44:20.251 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:20.251 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 3.1 M 2024-03-11T20:44:20.251 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 13 M 2024-03-11T20:44:20.251 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:44:20.263 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:20.675 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:20.993 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:20.993 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:20.993 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:44:20.994 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:44:20.994 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:20.994 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:44:21.271 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-mirror 2024-03-11T20:44:21.441 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-mirror-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 2.6 MB/s | 3.1 MB 00:01 2024-03-11T20:44:21.441 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:44:21.442 INFO:teuthology.orchestra.run.smithi129.stdout:Total 2.6 MB/s | 3.1 MB 00:01 2024-03-11T20:44:21.442 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:44:21.451 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:44:21.451 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:44:21.524 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:44:21.524 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:44:21.854 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:51 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:44:21.948 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:44:22.003 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:44:22.004 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:22.004 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-03-11T20:44:22.004 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:22.004 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:44:22.004 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.1 M 2024-03-11T20:44:22.004 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:22.004 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:44:22.004 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:22.004 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-11T20:44:22.005 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:22.005 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 3.1 M 2024-03-11T20:44:22.005 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 13 M 2024-03-11T20:44:22.005 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:44:22.016 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:22.041 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:22.042 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T20:44:22.042 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T20:44:22.042 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T20:44:22.042 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T20:44:22.042 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:22.993 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-mirror-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 3.1 MB/s | 3.1 MB 00:00 2024-03-11T20:44:22.994 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:44:22.994 INFO:teuthology.orchestra.run.smithi157.stdout:Total 3.1 MB/s | 3.1 MB 00:00 2024-03-11T20:44:22.994 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:44:23.004 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:44:23.004 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:44:23.077 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:44:23.077 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:44:23.212 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:23.212 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:23.212 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:44:23.212 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:44:23.212 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:23.212 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:44:23.463 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-nbd 2024-03-11T20:44:23.513 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:44:23.569 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:23.594 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:23.595 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T20:44:23.595 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T20:44:23.595 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T20:44:23.595 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T20:44:23.595 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:24.039 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:02:01 ago on Mon 11 Mar 2024 08:42:23 PM UTC. 2024-03-11T20:44:24.188 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-03-11T20:44:24.189 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:24.189 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-03-11T20:44:24.189 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:24.189 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-03-11T20:44:24.189 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 170 k 2024-03-11T20:44:24.189 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:24.189 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-03-11T20:44:24.189 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-03-11T20:44:24.189 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-03-11T20:44:24.189 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:24.190 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 170 k 2024-03-11T20:44:24.190 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 490 k 2024-03-11T20:44:24.190 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-03-11T20:44:24.537 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-nbd-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 491 kB/s | 170 kB 00:00 2024-03-11T20:44:24.537 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:44:24.538 INFO:teuthology.orchestra.run.smithi129.stdout:Total 489 kB/s | 170 kB 00:00 2024-03-11T20:44:24.538 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-03-11T20:44:24.545 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-03-11T20:44:24.546 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-03-11T20:44:24.601 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-03-11T20:44:24.601 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-03-11T20:44:24.735 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:24.735 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:24.735 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:44:24.735 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:44:24.735 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:24.735 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:44:24.749 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-03-11T20:44:24.844 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:25.004 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-nbd 2024-03-11T20:44:25.303 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:25.586 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:55 ago on Mon 11 Mar 2024 08:42:30 PM UTC. 2024-03-11T20:44:25.628 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:25.628 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:25.628 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-03-11T20:44:25.629 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:44:25.629 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:44:25.629 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-03-11T20:44:25.735 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-11T20:44:25.736 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:25.736 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-03-11T20:44:25.736 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:25.736 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-11T20:44:25.737 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 170 k 2024-03-11T20:44:25.737 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:25.737 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-11T20:44:25.737 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-11T20:44:25.737 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-03-11T20:44:25.737 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:25.737 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 170 k 2024-03-11T20:44:25.737 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 490 k 2024-03-11T20:44:25.737 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-11T20:44:25.889 DEBUG:teuthology.parallel:result is None 2024-03-11T20:44:26.072 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-nbd-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 509 kB/s | 170 kB 00:00 2024-03-11T20:44:26.072 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:44:26.073 INFO:teuthology.orchestra.run.smithi157.stdout:Total 507 kB/s | 170 kB 00:00 2024-03-11T20:44:26.073 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-11T20:44:26.081 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-11T20:44:26.081 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-11T20:44:26.137 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-11T20:44:26.137 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-11T20:44:26.286 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-11T20:44:26.395 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:26.881 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T20:44:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:27.186 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-11T20:44:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T20:44:27.186 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:44:27.186 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-11T20:44:27.433 DEBUG:teuthology.parallel:result is None 2024-03-11T20:44:27.434 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:44:27.657 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T20:44:27.682 INFO:teuthology.orchestra.run.smithi129.stdout:19.0.0-1979.g2b0101eb.el9 2024-03-11T20:44:27.682 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1979.g2b0101eb.el9 2024-03-11T20:44:27.683 INFO:teuthology.task.install:The correct ceph version 19.0.0-1979.g2b0101eb is installed. 2024-03-11T20:44:27.684 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:44:27.903 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T20:44:27.924 INFO:teuthology.orchestra.run.smithi157.stdout:19.0.0-1979.g2b0101eb.el9 2024-03-11T20:44:27.924 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1979.g2b0101eb.el9 2024-03-11T20:44:27.924 INFO:teuthology.task.install:The correct ceph version 19.0.0-1979.g2b0101eb is installed. 2024-03-11T20:44:27.926 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-11T20:44:27.926 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:44:27.926 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T20:44:27.958 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-11T20:44:27.958 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T20:44:27.991 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-11T20:44:27.992 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:44:27.992 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T20:44:28.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T20:44:28.099 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-11T20:44:28.099 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T20:44:28.126 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T20:44:28.194 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-11T20:44:28.194 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:44:28.194 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T20:44:28.223 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T20:44:28.292 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-11T20:44:28.292 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T20:44:28.318 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T20:44:28.386 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-11T20:44:28.386 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:44:28.386 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T20:44:28.415 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T20:44:28.483 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-11T20:44:28.484 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T20:44:28.511 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T20:44:28.578 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T20:44:28.706 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 1, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'cephadm_mode': 'root'} 2024-03-11T20:44:28.709 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:44:28.709 INFO:tasks.cephadm:Cluster fsid is 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:44:28.710 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T20:44:28.710 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.129', 'mon.c': '[v2:172.21.15.129:3301,v1:172.21.15.129:6790]', 'mon.b': '172.21.15.157'} 2024-03-11T20:44:28.710 INFO:tasks.cephadm:First mon is mon.a on smithi129 2024-03-11T20:44:28.710 INFO:tasks.cephadm:First mgr is y 2024-03-11T20:44:28.710 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T20:44:28.710 DEBUG:teuthology.orchestra.run.smithi129:> sudo hostname $(hostname -s) 2024-03-11T20:44:28.741 DEBUG:teuthology.orchestra.run.smithi157:> sudo hostname $(hostname -s) 2024-03-11T20:44:28.768 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-11T20:44:28.769 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:44:28.911 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54565/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1979-g2b0101eb', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1979.g2b0101eb'}, 'url': 'https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-11 00:47:00.272739', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-11T20:44:29.053 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref main, sha1 2b0101eb74d60610d544cb8a473e23a35ac1cc0b from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:44:29.055 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm 2024-03-11T20:44:29.055 DEBUG:teuthology.orchestra.run.smithi129:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T20:44:29.725 INFO:teuthology.orchestra.run.smithi129.stdout:-rw-r--r--. 1 ubuntu ubuntu 767461 Mar 11 20:44 /home/ubuntu/cephtest/cephadm 2024-03-11T20:44:29.725 DEBUG:teuthology.orchestra.run.smithi157:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T20:44:30.185 INFO:teuthology.orchestra.run.smithi157.stdout:-rw-r--r--. 1 ubuntu ubuntu 767461 Mar 11 20:44 /home/ubuntu/cephtest/cephadm 2024-03-11T20:44:30.185 DEBUG:teuthology.orchestra.run.smithi129:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T20:44:30.204 DEBUG:teuthology.orchestra.run.smithi157:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T20:44:30.235 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b on all hosts... 2024-03-11T20:44:30.235 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-11T20:44:30.246 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-11T20:44:31.165 INFO:teuthology.orchestra.run.smithi129.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T20:44:31.182 INFO:teuthology.orchestra.run.smithi157.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T20:45:01.167 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-03-11T20:45:01.167 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-11T20:45:01.167 INFO:teuthology.orchestra.run.smithi129.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-11T20:45:01.167 INFO:teuthology.orchestra.run.smithi129.stdout: "repo_digests": [ 2024-03-11T20:45:01.167 INFO:teuthology.orchestra.run.smithi129.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-11T20:45:01.167 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-03-11T20:45:01.167 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-03-11T20:45:17.842 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-03-11T20:45:17.842 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-11T20:45:17.842 INFO:teuthology.orchestra.run.smithi157.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-11T20:45:17.842 INFO:teuthology.orchestra.run.smithi157.stdout: "repo_digests": [ 2024-03-11T20:45:17.842 INFO:teuthology.orchestra.run.smithi157.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-11T20:45:17.843 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-03-11T20:45:17.843 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-03-11T20:45:17.867 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph 2024-03-11T20:45:17.903 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /etc/ceph 2024-03-11T20:45:17.933 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /etc/ceph 2024-03-11T20:45:17.974 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 777 /etc/ceph 2024-03-11T20:45:18.001 INFO:tasks.cephadm:Writing seed config... 2024-03-11T20:45:18.003 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T20:45:18.003 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T20:45:18.003 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-11T20:45:18.003 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-11T20:45:18.003 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-11T20:45:18.003 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T20:45:18.003 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T20:45:18.003 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T20:45:18.003 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-03-11T20:45:18.003 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T20:45:18.003 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T20:45:18.003 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-03-11T20:45:18.003 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-03-11T20:45:18.003 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-11T20:45:18.004 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:45:18.004 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T20:45:18.032 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 27bbebfa-dfe8-11ee-95c8-87774f69a715 mon election default strategy = 1 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2024-03-11T20:45:18.032 DEBUG:teuthology.orchestra.run.smithi129:mon.a> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mon.a.service 2024-03-11T20:45:18.074 DEBUG:teuthology.orchestra.run.smithi129:mgr.y> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mgr.y.service 2024-03-11T20:45:18.117 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T20:45:18.117 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b -v bootstrap --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.129 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T20:45:18.317 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-03-11T20:45:18.317 INFO:teuthology.orchestra.run.smithi129.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b', '-v', 'bootstrap', '--fsid', '27bbebfa-dfe8-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.129', '--skip-admin-label'] 2024-03-11T20:45:18.342 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T20:45:18.342 INFO:teuthology.orchestra.run.smithi129.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-11T20:45:18.343 INFO:teuthology.orchestra.run.smithi129.stdout:Verifying podman|docker is present... 2024-03-11T20:45:18.367 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T20:45:18.367 INFO:teuthology.orchestra.run.smithi129.stdout:Verifying lvm2 is present... 2024-03-11T20:45:18.368 INFO:teuthology.orchestra.run.smithi129.stdout:Verifying time synchronization is in place... 2024-03-11T20:45:18.375 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T20:45:18.375 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T20:45:18.381 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T20:45:18.381 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout inactive 2024-03-11T20:45:18.388 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout enabled 2024-03-11T20:45:18.394 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout active 2024-03-11T20:45:18.394 INFO:teuthology.orchestra.run.smithi129.stdout:Unit chronyd.service is enabled and running 2024-03-11T20:45:18.394 INFO:teuthology.orchestra.run.smithi129.stdout:Repeating the final host check... 2024-03-11T20:45:18.417 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T20:45:18.417 INFO:teuthology.orchestra.run.smithi129.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-11T20:45:18.417 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl is present 2024-03-11T20:45:18.418 INFO:teuthology.orchestra.run.smithi129.stdout:lvcreate is present 2024-03-11T20:45:18.424 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T20:45:18.424 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T20:45:18.430 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T20:45:18.430 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout inactive 2024-03-11T20:45:18.436 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout enabled 2024-03-11T20:45:18.443 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout active 2024-03-11T20:45:18.443 INFO:teuthology.orchestra.run.smithi129.stdout:Unit chronyd.service is enabled and running 2024-03-11T20:45:18.443 INFO:teuthology.orchestra.run.smithi129.stdout:Host looks OK 2024-03-11T20:45:18.443 INFO:teuthology.orchestra.run.smithi129.stdout:Cluster fsid: 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:45:18.443 INFO:teuthology.orchestra.run.smithi129.stdout:Acquiring lock 140550619676096 on /run/cephadm/27bbebfa-dfe8-11ee-95c8-87774f69a715.lock 2024-03-11T20:45:18.443 INFO:teuthology.orchestra.run.smithi129.stdout:Lock 140550619676096 acquired on /run/cephadm/27bbebfa-dfe8-11ee-95c8-87774f69a715.lock 2024-03-11T20:45:18.443 INFO:teuthology.orchestra.run.smithi129.stdout:Verifying IP 172.21.15.129 port 3300 ... 2024-03-11T20:45:18.444 INFO:teuthology.orchestra.run.smithi129.stdout:Verifying IP 172.21.15.129 port 6789 ... 2024-03-11T20:45:18.444 INFO:teuthology.orchestra.run.smithi129.stdout:Base mon IP(s) is [172.21.15.129:3300, 172.21.15.129:6789], mon addrv is [v2:172.21.15.129:3300,v1:172.21.15.129:6789] 2024-03-11T20:45:18.447 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.129 metric 100 2024-03-11T20:45:18.447 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.129 metric 100 2024-03-11T20:45:18.450 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T20:45:18.450 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-11T20:45:18.450 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1793sec hoplimit 64 pref medium 2024-03-11T20:45:18.452 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T20:45:18.452 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-11T20:45:18.452 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T20:45:18.452 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-11T20:45:18.452 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:7261/64 scope link 2024-03-11T20:45:18.452 INFO:teuthology.orchestra.run.smithi129.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T20:45:18.453 INFO:teuthology.orchestra.run.smithi129.stdout:Mon IP `172.21.15.129` is in CIDR network `172.21.0.0/20` 2024-03-11T20:45:18.453 INFO:teuthology.orchestra.run.smithi129.stdout:Mon IP `172.21.15.129` is in CIDR network `172.21.0.0/20` 2024-03-11T20:45:18.453 INFO:teuthology.orchestra.run.smithi129.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-11T20:45:18.453 INFO:teuthology.orchestra.run.smithi129.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T20:45:18.454 INFO:teuthology.orchestra.run.smithi129.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T20:45:21.409 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stdout 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-11T20:45:21.410 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T20:45:21.410 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stderr Getting image source signatures 2024-03-11T20:45:21.410 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stderr Copying blob sha256:4392e0adce526b0883d65bbedc13478c05c0e708702f1c7015b93fe77c162cd5 2024-03-11T20:45:21.411 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-11T20:45:21.411 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stderr Copying config sha256:4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-11T20:45:21.411 INFO:teuthology.orchestra.run.smithi129.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-11T20:45:25.952 INFO:teuthology.orchestra.run.smithi129.stdout:ceph: stdout ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T20:45:25.952 INFO:teuthology.orchestra.run.smithi129.stdout:Ceph version: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T20:45:25.952 INFO:teuthology.orchestra.run.smithi129.stdout:Extracting ceph user uid/gid from container image... 2024-03-11T20:45:26.868 INFO:teuthology.orchestra.run.smithi129.stdout:stat: stdout 167 167 2024-03-11T20:45:26.868 INFO:teuthology.orchestra.run.smithi129.stdout:Creating initial keys... 2024-03-11T20:45:27.820 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph-authtool: stdout AQBnbe9lPqLSDhAAoQW8lReAJB1MPFuZBu1iWA== 2024-03-11T20:45:29.443 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph-authtool: stdout AQBobe9lXNClDBAAk9bz2exFK+uZQ4SPHP5H2A== 2024-03-11T20:45:30.389 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph-authtool: stdout AQBpbe9lMBq7MBAAz/eIWfeypi9Rh4vT5KqkVA== 2024-03-11T20:45:30.389 INFO:teuthology.orchestra.run.smithi129.stdout:Creating initial monmap... 2024-03-11T20:45:32.787 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T20:45:32.787 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-11T20:45:32.787 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:45:32.787 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T20:45:32.787 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool for a [v2:172.21.15.129:3300,v1:172.21.15.129:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T20:45:32.787 INFO:teuthology.orchestra.run.smithi129.stdout:setting min_mon_release = quincy 2024-03-11T20:45:32.787 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/monmaptool: set fsid to 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:45:32.788 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T20:45:32.788 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:45:32.788 INFO:teuthology.orchestra.run.smithi129.stdout:Creating mon... 2024-03-11T20:45:33.792 INFO:teuthology.orchestra.run.smithi129.stdout:create mon.a on 2024-03-11T20:45:33.987 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-11T20:45:34.119 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-11T20:45:34.297 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715.target. 2024-03-11T20:45:34.297 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715.target. 2024-03-11T20:45:34.520 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mon.a 2024-03-11T20:45:34.520 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Failed to reset failed state of unit ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-11T20:45:34.681 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715.target.wants/ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@.service. 2024-03-11T20:45:35.216 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T20:45:35.217 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout disabled 2024-03-11T20:45:35.223 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T20:45:35.223 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout inactive 2024-03-11T20:45:35.223 INFO:teuthology.orchestra.run.smithi129.stdout:firewalld.service is not enabled 2024-03-11T20:45:35.223 INFO:teuthology.orchestra.run.smithi129.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T20:45:35.224 INFO:teuthology.orchestra.run.smithi129.stdout:Waiting for mon to start... 2024-03-11T20:45:35.224 INFO:teuthology.orchestra.run.smithi129.stdout:Waiting for mon... 2024-03-11T20:45:36.301 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:36 smithi129 ceph-mon[29695]: from='client.? 172.21.15.129:0/1622119573' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-11T20:45:36.309 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout cluster: 2024-03-11T20:45:36.309 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout id: 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:45:36.309 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-11T20:45:36.309 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-03-11T20:45:36.310 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout services: 2024-03-11T20:45:36.310 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.588249s) 2024-03-11T20:45:36.310 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-11T20:45:36.310 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-11T20:45:36.310 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-03-11T20:45:36.310 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout data: 2024-03-11T20:45:36.310 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-11T20:45:36.310 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-11T20:45:36.310 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-11T20:45:36.310 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout pgs: 2024-03-11T20:45:36.310 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-03-11T20:45:36.310 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.678+0000 7fd0b5118700 1 Processor -- start 2024-03-11T20:45:36.310 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.679+0000 7fd0b5118700 1 -- start start 2024-03-11T20:45:36.311 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.679+0000 7fd0b5118700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0b0104b60 0x7fd0b0104f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:36.311 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.679+0000 7fd0b5118700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b0105550 con 0x7fd0b0104b60 2024-03-11T20:45:36.311 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.679+0000 7fd0aed9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0b0104b60 0x7fd0b0104f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:36.311 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.679+0000 7fd0aed9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0b0104b60 0x7fd0b0104f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36994/0 (socket says 172.21.15.129:36994) 2024-03-11T20:45:36.311 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.679+0000 7fd0aed9d700 1 -- 172.21.15.129:0/338051983 learned_addr learned my addr 172.21.15.129:0/338051983 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:36.311 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.679+0000 7fd0aed9d700 1 -- 172.21.15.129:0/338051983 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b0105d60 con 0x7fd0b0104b60 2024-03-11T20:45:36.311 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.679+0000 7fd0aed9d700 1 --2- 172.21.15.129:0/338051983 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0b0104b60 0x7fd0b0104f80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800bb40 tx=0x7fd0a800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade3e1986052ff24 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:36.311 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.680+0000 7fd0add9b700 1 -- 172.21.15.129:0/338051983 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0a800dbe0 con 0x7fd0b0104b60 2024-03-11T20:45:36.311 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.680+0000 7fd0add9b700 1 -- 172.21.15.129:0/338051983 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a800dd80 con 0x7fd0b0104b60 2024-03-11T20:45:36.311 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.680+0000 7fd0add9b700 1 -- 172.21.15.129:0/338051983 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0a8004180 con 0x7fd0b0104b60 2024-03-11T20:45:36.311 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.680+0000 7fd0b5118700 1 -- 172.21.15.129:0/338051983 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0b0104b60 msgr2=0x7fd0b0104f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:36.311 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.680+0000 7fd0b5118700 1 --2- 172.21.15.129:0/338051983 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0b0104b60 0x7fd0b0104f80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800bb40 tx=0x7fd0a800be50 comp rx=0 tx=0).stop 2024-03-11T20:45:36.311 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.680+0000 7fd0b5118700 1 -- 172.21.15.129:0/338051983 shutdown_connections 2024-03-11T20:45:36.312 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.680+0000 7fd0b5118700 1 --2- 172.21.15.129:0/338051983 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0b0104b60 0x7fd0b0104f80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:36.312 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.680+0000 7fd0b5118700 1 -- 172.21.15.129:0/338051983 >> 172.21.15.129:0/338051983 conn(0x7fd0b0100120 msgr2=0x7fd0b0102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:36.312 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.680+0000 7fd0b5118700 1 -- 172.21.15.129:0/338051983 shutdown_connections 2024-03-11T20:45:36.312 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.680+0000 7fd0b5118700 1 -- 172.21.15.129:0/338051983 wait complete. 2024-03-11T20:45:36.312 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.681+0000 7fd0b5118700 1 Processor -- start 2024-03-11T20:45:36.312 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.681+0000 7fd0b5118700 1 -- start start 2024-03-11T20:45:36.312 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.681+0000 7fd0b5118700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0b0104b60 0x7fd0b01970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:36.312 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.681+0000 7fd0b5118700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b01061d0 con 0x7fd0b0104b60 2024-03-11T20:45:36.312 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.681+0000 7fd0aed9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0b0104b60 0x7fd0b01970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:36.312 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.682+0000 7fd0aed9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0b0104b60 0x7fd0b01970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37002/0 (socket says 172.21.15.129:37002) 2024-03-11T20:45:36.312 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.682+0000 7fd0aed9d700 1 -- 172.21.15.129:0/1622119573 learned_addr learned my addr 172.21.15.129:0/1622119573 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:36.312 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.682+0000 7fd0aed9d700 1 -- 172.21.15.129:0/1622119573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b01975f0 con 0x7fd0b0104b60 2024-03-11T20:45:36.312 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.682+0000 7fd0aed9d700 1 --2- 172.21.15.129:0/1622119573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0b0104b60 0x7fd0b01970b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800b340 tx=0x7fd0a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:36.312 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.682+0000 7fd0a7fff700 1 -- 172.21.15.129:0/1622119573 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0a8029070 con 0x7fd0b0104b60 2024-03-11T20:45:36.313 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.682+0000 7fd0a7fff700 1 -- 172.21.15.129:0/1622119573 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a80044e0 con 0x7fd0b0104b60 2024-03-11T20:45:36.313 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.682+0000 7fd0b5118700 1 -- 172.21.15.129:0/1622119573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b0197880 con 0x7fd0b0104b60 2024-03-11T20:45:36.313 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.682+0000 7fd0a7fff700 1 -- 172.21.15.129:0/1622119573 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0a80105a0 con 0x7fd0b0104b60 2024-03-11T20:45:36.313 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.682+0000 7fd0b5118700 1 -- 172.21.15.129:0/1622119573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b019a570 con 0x7fd0b0104b60 2024-03-11T20:45:36.313 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.682+0000 7fd0a7fff700 1 -- 172.21.15.129:0/1622119573 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd0a802c070 con 0x7fd0b0104b60 2024-03-11T20:45:36.313 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.682+0000 7fd0a7fff700 1 -- 172.21.15.129:0/1622119573 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd0a8021070 con 0x7fd0b0104b60 2024-03-11T20:45:36.313 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.683+0000 7fd0b5118700 1 -- 172.21.15.129:0/1622119573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b0063410 con 0x7fd0b0104b60 2024-03-11T20:45:36.313 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.685+0000 7fd0a7fff700 1 -- 172.21.15.129:0/1622119573 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd0a8015070 con 0x7fd0b0104b60 2024-03-11T20:45:36.313 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.757+0000 7fd0b5118700 1 -- 172.21.15.129:0/1622119573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fd0b0197d60 con 0x7fd0b0104b60 2024-03-11T20:45:36.313 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.757+0000 7fd0a7fff700 1 -- 172.21.15.129:0/1622119573 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fd0b0197d60 con 0x7fd0b0104b60 2024-03-11T20:45:36.313 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.758+0000 7fd0b5118700 1 -- 172.21.15.129:0/1622119573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0b0104b60 msgr2=0x7fd0b01970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:36.314 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.758+0000 7fd0b5118700 1 --2- 172.21.15.129:0/1622119573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0b0104b60 0x7fd0b01970b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd0a800b340 tx=0x7fd0a8004750 comp rx=0 tx=0).stop 2024-03-11T20:45:36.314 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.758+0000 7fd0b5118700 1 -- 172.21.15.129:0/1622119573 shutdown_connections 2024-03-11T20:45:36.314 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.758+0000 7fd0b5118700 1 --2- 172.21.15.129:0/1622119573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0b0104b60 0x7fd0b01970b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:36.314 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.758+0000 7fd0b5118700 1 -- 172.21.15.129:0/1622119573 >> 172.21.15.129:0/1622119573 conn(0x7fd0b0100120 msgr2=0x7fd0b018dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:36.314 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.758+0000 7fd0b5118700 1 -- 172.21.15.129:0/1622119573 shutdown_connections 2024-03-11T20:45:36.314 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:35.758+0000 7fd0b5118700 1 -- 172.21.15.129:0/1622119573 wait complete. 2024-03-11T20:45:36.314 INFO:teuthology.orchestra.run.smithi129.stdout:mon is available 2024-03-11T20:45:36.314 INFO:teuthology.orchestra.run.smithi129.stdout:Assimilating anything we can from ceph.conf... 2024-03-11T20:45:37.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:37 smithi129 ceph-mon[29695]: from='client.? 172.21.15.129:0/589364221' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T20:45:37.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:37 smithi129 ceph-mon[29695]: from='client.? 172.21.15.129:0/589364221' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T20:45:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-03-11T20:45:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T20:45:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout fsid = 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:45:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.129:3300,v1:172.21.15.129:6789] 2024-03-11T20:45:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T20:45:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T20:45:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T20:45:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T20:45:37.926 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-03-11T20:45:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T20:45:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T20:45:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-03-11T20:45:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T20:45:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T20:45:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T20:45:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.770+0000 7f6bee5fa700 1 Processor -- start 2024-03-11T20:45:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.770+0000 7f6bee5fa700 1 -- start start 2024-03-11T20:45:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.771+0000 7f6bee5fa700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6be8106110 0x7f6be8106530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.771+0000 7f6bee5fa700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be8106b00 con 0x7f6be8106110 2024-03-11T20:45:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.771+0000 7f6be7fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6be8106110 0x7f6be8106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.771+0000 7f6be7fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6be8106110 0x7f6be8106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37018/0 (socket says 172.21.15.129:37018) 2024-03-11T20:45:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.771+0000 7f6be7fff700 1 -- 172.21.15.129:0/3375481810 learned_addr learned my addr 172.21.15.129:0/3375481810 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:37.927 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.771+0000 7f6be7fff700 1 -- 172.21.15.129:0/3375481810 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be8104d20 con 0x7f6be8106110 2024-03-11T20:45:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.771+0000 7f6be7fff700 1 --2- 172.21.15.129:0/3375481810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6be8106110 0x7f6be8106530 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6bd80098a0 tx=0x7f6bd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1629ab443aafe7b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.771+0000 7f6be6ffd700 1 -- 172.21.15.129:0/3375481810 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6bd8004160 con 0x7f6be8106110 2024-03-11T20:45:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.771+0000 7f6be6ffd700 1 -- 172.21.15.129:0/3375481810 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd8004300 con 0x7f6be8106110 2024-03-11T20:45:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.772+0000 7f6be6ffd700 1 -- 172.21.15.129:0/3375481810 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6bd80036a0 con 0x7f6be8106110 2024-03-11T20:45:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.772+0000 7f6bee5fa700 1 -- 172.21.15.129:0/3375481810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6be8106110 msgr2=0x7f6be8106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.772+0000 7f6bee5fa700 1 --2- 172.21.15.129:0/3375481810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6be8106110 0x7f6be8106530 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6bd80098a0 tx=0x7f6bd8009bb0 comp rx=0 tx=0).stop 2024-03-11T20:45:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.772+0000 7f6bee5fa700 1 -- 172.21.15.129:0/3375481810 shutdown_connections 2024-03-11T20:45:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.772+0000 7f6bee5fa700 1 --2- 172.21.15.129:0/3375481810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6be8106110 0x7f6be8106530 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.772+0000 7f6bee5fa700 1 -- 172.21.15.129:0/3375481810 >> 172.21.15.129:0/3375481810 conn(0x7f6be80ff850 msgr2=0x7f6be8101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.772+0000 7f6bee5fa700 1 -- 172.21.15.129:0/3375481810 shutdown_connections 2024-03-11T20:45:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.772+0000 7f6bee5fa700 1 -- 172.21.15.129:0/3375481810 wait complete. 2024-03-11T20:45:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.773+0000 7f6bee5fa700 1 Processor -- start 2024-03-11T20:45:37.928 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.773+0000 7f6bee5fa700 1 -- start start 2024-03-11T20:45:37.929 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.773+0000 7f6bee5fa700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6be8106110 0x7f6be81992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:37.929 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.773+0000 7f6bee5fa700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be8105250 con 0x7f6be8106110 2024-03-11T20:45:37.929 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.773+0000 7f6be7fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6be8106110 0x7f6be81992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:37.929 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.773+0000 7f6be7fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6be8106110 0x7f6be81992d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37032/0 (socket says 172.21.15.129:37032) 2024-03-11T20:45:37.929 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.773+0000 7f6be7fff700 1 -- 172.21.15.129:0/589364221 learned_addr learned my addr 172.21.15.129:0/589364221 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:37.929 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.773+0000 7f6be7fff700 1 -- 172.21.15.129:0/589364221 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be8199810 con 0x7f6be8106110 2024-03-11T20:45:37.929 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.774+0000 7f6be7fff700 1 --2- 172.21.15.129:0/589364221 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6be8106110 0x7f6be81992d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8000c00 tx=0x7f6bd8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:37.929 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.774+0000 7f6be57fa700 1 -- 172.21.15.129:0/589364221 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6bd8003db0 con 0x7f6be8106110 2024-03-11T20:45:37.929 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.774+0000 7f6be57fa700 1 -- 172.21.15.129:0/589364221 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd8011630 con 0x7f6be8106110 2024-03-11T20:45:37.929 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.774+0000 7f6bee5fa700 1 -- 172.21.15.129:0/589364221 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be8199aa0 con 0x7f6be8106110 2024-03-11T20:45:37.929 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.774+0000 7f6be57fa700 1 -- 172.21.15.129:0/589364221 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6bd8011950 con 0x7f6be8106110 2024-03-11T20:45:37.929 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.774+0000 7f6bee5fa700 1 -- 172.21.15.129:0/589364221 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be819c790 con 0x7f6be8106110 2024-03-11T20:45:37.929 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.775+0000 7f6be57fa700 1 -- 172.21.15.129:0/589364221 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6bd801a460 con 0x7f6be8106110 2024-03-11T20:45:37.929 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.775+0000 7f6be57fa700 1 -- 172.21.15.129:0/589364221 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6bd801b780 con 0x7f6be8106110 2024-03-11T20:45:37.930 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.775+0000 7f6bee5fa700 1 -- 172.21.15.129:0/589364221 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be8063410 con 0x7f6be8106110 2024-03-11T20:45:37.930 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.777+0000 7f6be57fa700 1 -- 172.21.15.129:0/589364221 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6bd8031020 con 0x7f6be8106110 2024-03-11T20:45:37.930 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.846+0000 7f6bee5fa700 1 -- 172.21.15.129:0/589364221 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f6be802d8d0 con 0x7f6be8106110 2024-03-11T20:45:37.930 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.849+0000 7f6be57fa700 1 -- 172.21.15.129:0/589364221 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+340 (secure 0 0 0) 0x7f6bd801b9d0 con 0x7f6be8106110 2024-03-11T20:45:37.930 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.850+0000 7f6be57fa700 1 -- 172.21.15.129:0/589364221 <== mon.0 v2:172.21.15.129:3300/0 8 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f6bd801aca0 con 0x7f6be8106110 2024-03-11T20:45:37.930 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.851+0000 7f6bee5fa700 1 -- 172.21.15.129:0/589364221 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6be8106110 msgr2=0x7f6be81992d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:37.930 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.851+0000 7f6bee5fa700 1 --2- 172.21.15.129:0/589364221 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6be8106110 0x7f6be81992d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8000c00 tx=0x7f6bd8004750 comp rx=0 tx=0).stop 2024-03-11T20:45:37.930 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.851+0000 7f6bee5fa700 1 -- 172.21.15.129:0/589364221 shutdown_connections 2024-03-11T20:45:37.930 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.851+0000 7f6bee5fa700 1 --2- 172.21.15.129:0/589364221 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6be8106110 0x7f6be81992d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:37.930 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.851+0000 7f6bee5fa700 1 -- 172.21.15.129:0/589364221 >> 172.21.15.129:0/589364221 conn(0x7f6be80ff850 msgr2=0x7f6be818fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:37.930 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.851+0000 7f6bee5fa700 1 -- 172.21.15.129:0/589364221 shutdown_connections 2024-03-11T20:45:37.930 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:36.851+0000 7f6bee5fa700 1 -- 172.21.15.129:0/589364221 wait complete. 2024-03-11T20:45:37.930 INFO:teuthology.orchestra.run.smithi129.stdout:Generating new minimal ceph.conf... 2024-03-11T20:45:38.713 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:38 smithi129 ceph-mon[29695]: from='client.? 172.21.15.129:0/3617163500' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:45:39.013 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.343+0000 7f32179b5700 1 Processor -- start 2024-03-11T20:45:39.013 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.343+0000 7f32179b5700 1 -- start start 2024-03-11T20:45:39.013 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.343+0000 7f32179b5700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3210103f50 0x7f3210104370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:39.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.343+0000 7f32179b5700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3210104940 con 0x7f3210103f50 2024-03-11T20:45:39.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.343+0000 7f3215751700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3210103f50 0x7f3210104370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:39.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.343+0000 7f3215751700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3210103f50 0x7f3210104370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37036/0 (socket says 172.21.15.129:37036) 2024-03-11T20:45:39.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.343+0000 7f3215751700 1 -- 172.21.15.129:0/960866710 learned_addr learned my addr 172.21.15.129:0/960866710 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:39.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.344+0000 7f3215751700 1 -- 172.21.15.129:0/960866710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3210104ac0 con 0x7f3210103f50 2024-03-11T20:45:39.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.344+0000 7f3215751700 1 --2- 172.21.15.129:0/960866710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3210103f50 0x7f3210104370 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f320c0098a0 tx=0x7f320c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d27b5443f3dd974 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:39.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.344+0000 7f3207fff700 1 -- 172.21.15.129:0/960866710 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f320c004160 con 0x7f3210103f50 2024-03-11T20:45:39.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.344+0000 7f3207fff700 1 -- 172.21.15.129:0/960866710 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f320c004300 con 0x7f3210103f50 2024-03-11T20:45:39.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.344+0000 7f3207fff700 1 -- 172.21.15.129:0/960866710 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f320c011940 con 0x7f3210103f50 2024-03-11T20:45:39.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.345+0000 7f32179b5700 1 -- 172.21.15.129:0/960866710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3210103f50 msgr2=0x7f3210104370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:39.014 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.345+0000 7f32179b5700 1 --2- 172.21.15.129:0/960866710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3210103f50 0x7f3210104370 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f320c0098a0 tx=0x7f320c009bb0 comp rx=0 tx=0).stop 2024-03-11T20:45:39.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.345+0000 7f32179b5700 1 -- 172.21.15.129:0/960866710 shutdown_connections 2024-03-11T20:45:39.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.345+0000 7f32179b5700 1 --2- 172.21.15.129:0/960866710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3210103f50 0x7f3210104370 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:39.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.345+0000 7f32179b5700 1 -- 172.21.15.129:0/960866710 >> 172.21.15.129:0/960866710 conn(0x7f32100ff870 msgr2=0x7f3210101cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:39.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.345+0000 7f32179b5700 1 -- 172.21.15.129:0/960866710 shutdown_connections 2024-03-11T20:45:39.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.345+0000 7f32179b5700 1 -- 172.21.15.129:0/960866710 wait complete. 2024-03-11T20:45:39.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.345+0000 7f32179b5700 1 Processor -- start 2024-03-11T20:45:39.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.345+0000 7f32179b5700 1 -- start start 2024-03-11T20:45:39.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.345+0000 7f32179b5700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3210103f50 0x7f32101999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:39.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.345+0000 7f32179b5700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3210107c80 con 0x7f3210103f50 2024-03-11T20:45:39.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.346+0000 7f3215751700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3210103f50 0x7f32101999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:39.015 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.346+0000 7f3215751700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3210103f50 0x7f32101999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37048/0 (socket says 172.21.15.129:37048) 2024-03-11T20:45:39.016 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.346+0000 7f3215751700 1 -- 172.21.15.129:0/3617163500 learned_addr learned my addr 172.21.15.129:0/3617163500 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:39.016 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.346+0000 7f3215751700 1 -- 172.21.15.129:0/3617163500 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3210199f20 con 0x7f3210103f50 2024-03-11T20:45:39.016 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.346+0000 7f3215751700 1 --2- 172.21.15.129:0/3617163500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3210103f50 0x7f32101999e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f320c009fb0 tx=0x7f320c0119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:39.016 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.346+0000 7f32067fc700 1 -- 172.21.15.129:0/3617163500 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f320c01a430 con 0x7f3210103f50 2024-03-11T20:45:39.016 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.346+0000 7f32067fc700 1 -- 172.21.15.129:0/3617163500 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f320c01aa30 con 0x7f3210103f50 2024-03-11T20:45:39.016 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.346+0000 7f32179b5700 1 -- 172.21.15.129:0/3617163500 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321019a210 con 0x7f3210103f50 2024-03-11T20:45:39.016 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.346+0000 7f32067fc700 1 -- 172.21.15.129:0/3617163500 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f320c024860 con 0x7f3210103f50 2024-03-11T20:45:39.016 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.346+0000 7f32179b5700 1 -- 172.21.15.129:0/3617163500 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321019cea0 con 0x7f3210103f50 2024-03-11T20:45:39.016 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.346+0000 7f32067fc700 1 -- 172.21.15.129:0/3617163500 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f320c022020 con 0x7f3210103f50 2024-03-11T20:45:39.016 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.346+0000 7f32067fc700 1 -- 172.21.15.129:0/3617163500 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f320c02d3f0 con 0x7f3210103f50 2024-03-11T20:45:39.016 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.347+0000 7f32179b5700 1 -- 172.21.15.129:0/3617163500 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3210063410 con 0x7f3210103f50 2024-03-11T20:45:39.016 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.349+0000 7f32067fc700 1 -- 172.21.15.129:0/3617163500 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f320c01abe0 con 0x7f3210103f50 2024-03-11T20:45:39.017 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.416+0000 7f32179b5700 1 -- 172.21.15.129:0/3617163500 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f321019d380 con 0x7f3210103f50 2024-03-11T20:45:39.017 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.417+0000 7f32067fc700 1 -- 172.21.15.129:0/3617163500 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f320c01a5d0 con 0x7f3210103f50 2024-03-11T20:45:39.017 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.418+0000 7f32179b5700 1 -- 172.21.15.129:0/3617163500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3210103f50 msgr2=0x7f32101999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:39.017 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.418+0000 7f32179b5700 1 --2- 172.21.15.129:0/3617163500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3210103f50 0x7f32101999e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f320c009fb0 tx=0x7f320c0119c0 comp rx=0 tx=0).stop 2024-03-11T20:45:39.017 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.418+0000 7f32179b5700 1 -- 172.21.15.129:0/3617163500 shutdown_connections 2024-03-11T20:45:39.017 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.418+0000 7f32179b5700 1 --2- 172.21.15.129:0/3617163500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3210103f50 0x7f32101999e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:39.017 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.418+0000 7f32179b5700 1 -- 172.21.15.129:0/3617163500 >> 172.21.15.129:0/3617163500 conn(0x7f32100ff870 msgr2=0x7f32101003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:39.017 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.418+0000 7f32179b5700 1 -- 172.21.15.129:0/3617163500 shutdown_connections 2024-03-11T20:45:39.017 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:38.418+0000 7f32179b5700 1 -- 172.21.15.129:0/3617163500 wait complete. 2024-03-11T20:45:39.018 INFO:teuthology.orchestra.run.smithi129.stdout:Restarting the monitor... 2024-03-11T20:45:39.262 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:39 smithi129 systemd[1]: Stopping Ceph mon.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T20:45:39.584 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:39 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-a[29672]: 2024-03-11T20:45:39.267+0000 7fbc1b32a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T20:45:39.585 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:39 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-a[29672]: 2024-03-11T20:45:39.267+0000 7fbc1b32a700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T20:45:39.585 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:39 smithi129 podman[30027]: 2024-03-11 20:45:39.339077501 +0000 UTC m=+0.156902724 container died 5ae6bbe3b8df4d8d19586b0ef5c7cce239d403e31f54361be9ab085f72e2bf40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-11T20:45:39.893 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:39 smithi129 podman[30027]: 2024-03-11 20:45:39.681501158 +0000 UTC m=+0.499326397 container cleanup 5ae6bbe3b8df4d8d19586b0ef5c7cce239d403e31f54361be9ab085f72e2bf40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-11T20:45:39.893 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:39 smithi129 bash[30027]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-a 2024-03-11T20:45:39.893 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:39 smithi129 podman[30045]: 2024-03-11 20:45:39.815675032 +0000 UTC m=+0.476046699 container remove 5ae6bbe3b8df4d8d19586b0ef5c7cce239d403e31f54361be9ab085f72e2bf40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T20:45:40.218 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T20:45:40.218 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 systemd[1]: Stopped Ceph mon.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T20:45:40.219 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 systemd[1]: Starting Ceph mon.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T20:45:40.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 podman[30146]: 2024-03-11 20:45:40.224810029 +0000 UTC m=+0.110619301 container create 4970c2d909d20e2a6c4a3d04b81de9dea01ea397a4addf67c6c955c886605dec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-a, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-11T20:45:40.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 podman[30146]: 2024-03-11 20:45:40.131876539 +0000 UTC m=+0.017685825 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:45:40.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 podman[30146]: 2024-03-11 20:45:40.468851803 +0000 UTC m=+0.354661090 container init 4970c2d909d20e2a6c4a3d04b81de9dea01ea397a4addf67c6c955c886605dec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=main-2b0101e, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-11T20:45:40.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 podman[30146]: 2024-03-11 20:45:40.473976278 +0000 UTC m=+0.359785550 container start 4970c2d909d20e2a6c4a3d04b81de9dea01ea397a4addf67c6c955c886605dec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-03-11T20:45:40.575 INFO:teuthology.orchestra.run.smithi129.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-11T20:45:40.800 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T20:45:40.800 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 7 2024-03-11T20:45:40.801 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: pidfile_write: ignore empty --pid-file 2024-03-11T20:45:40.801 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: load: jerasure load: lrc 2024-03-11T20:45:40.801 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: RocksDB version: 7.9.2 2024-03-11T20:45:40.801 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Git sha 0 2024-03-11T20:45:40.801 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-11T20:45:40.801 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: DB SUMMARY 2024-03-11T20:45:40.801 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: DB Session ID: G9MQU4N0FZLRMLP1RYR9 2024-03-11T20:45:40.801 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: CURRENT file: CURRENT 2024-03-11T20:45:40.801 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T20:45:40.801 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-11T20:45:40.801 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T20:45:40.801 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 86839 ; 2024-03-11T20:45:40.802 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.error_if_exists: 0 2024-03-11T20:45:40.802 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.create_if_missing: 0 2024-03-11T20:45:40.802 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.paranoid_checks: 1 2024-03-11T20:45:40.802 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T20:45:40.802 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T20:45:40.802 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T20:45:40.802 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.env: 0x563767bd7b20 2024-03-11T20:45:40.802 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T20:45:40.802 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.info_log: 0x56376924d8e0 2024-03-11T20:45:40.802 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T20:45:40.802 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.statistics: (nil) 2024-03-11T20:45:40.802 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.use_fsync: 0 2024-03-11T20:45:40.803 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_log_file_size: 0 2024-03-11T20:45:40.803 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T20:45:40.803 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T20:45:40.803 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T20:45:40.803 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T20:45:40.803 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.allow_fallocate: 1 2024-03-11T20:45:40.803 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T20:45:40.803 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T20:45:40.804 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.use_direct_reads: 0 2024-03-11T20:45:40.804 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T20:45:40.804 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T20:45:40.804 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.db_log_dir: 2024-03-11T20:45:40.804 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.wal_dir: 2024-03-11T20:45:40.804 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T20:45:40.805 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T20:45:40.805 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T20:45:40.805 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T20:45:40.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T20:45:40.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T20:45:40.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T20:45:40.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T20:45:40.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.write_buffer_manager: 0x563769221400 2024-03-11T20:45:40.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T20:45:40.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T20:45:40.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T20:45:40.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T20:45:40.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T20:45:40.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T20:45:40.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T20:45:40.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T20:45:40.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.unordered_write: 0 2024-03-11T20:45:40.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T20:45:40.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T20:45:40.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T20:45:40.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T20:45:40.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.row_cache: None 2024-03-11T20:45:40.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.wal_filter: None 2024-03-11T20:45:40.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T20:45:40.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T20:45:40.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.two_write_queues: 0 2024-03-11T20:45:40.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T20:45:40.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.wal_compression: 0 2024-03-11T20:45:40.807 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.atomic_flush: 0 2024-03-11T20:45:40.808 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T20:45:40.808 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T20:45:40.808 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T20:45:40.808 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.log_readahead_size: 0 2024-03-11T20:45:40.808 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T20:45:40.808 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T20:45:40.808 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T20:45:40.808 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T20:45:40.808 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T20:45:40.808 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T20:45:40.808 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T20:45:40.808 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_background_jobs: 2 2024-03-11T20:45:40.808 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_background_compactions: -1 2024-03-11T20:45:40.809 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_subcompactions: 1 2024-03-11T20:45:40.809 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T20:45:40.809 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T20:45:40.809 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T20:45:40.809 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T20:45:40.809 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T20:45:40.809 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T20:45:40.809 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T20:45:40.809 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T20:45:40.809 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_open_files: -1 2024-03-11T20:45:40.809 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T20:45:40.809 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T20:45:40.809 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T20:45:40.810 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T20:45:40.810 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_background_flushes: -1 2024-03-11T20:45:40.810 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Compression algorithms supported: 2024-03-11T20:45:40.810 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T20:45:40.810 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: kZSTD supported: 0 2024-03-11T20:45:40.810 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: kXpressCompression supported: 0 2024-03-11T20:45:40.810 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T20:45:40.810 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: kZlibCompression supported: 1 2024-03-11T20:45:40.810 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: kSnappyCompression supported: 1 2024-03-11T20:45:40.810 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: kLZ4Compression supported: 1 2024-03-11T20:45:40.810 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: kBZip2Compression supported: 0 2024-03-11T20:45:40.810 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T20:45:40.810 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T20:45:40.811 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-11T20:45:40.811 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T20:45:40.811 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T20:45:40.811 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.merge_operator: 2024-03-11T20:45:40.811 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compaction_filter: None 2024-03-11T20:45:40.811 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T20:45:40.811 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T20:45:40.811 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T20:45:40.811 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T20:45:40.811 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56376924d9e0) 2024-03-11T20:45:40.811 INFO:journalctl@ceph.mon.a.smithi129.stdout: cache_index_and_filter_blocks: 1 2024-03-11T20:45:40.811 INFO:journalctl@ceph.mon.a.smithi129.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T20:45:40.811 INFO:journalctl@ceph.mon.a.smithi129.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T20:45:40.811 INFO:journalctl@ceph.mon.a.smithi129.stdout: pin_top_level_index_and_filter: 1 2024-03-11T20:45:40.812 INFO:journalctl@ceph.mon.a.smithi129.stdout: index_type: 0 2024-03-11T20:45:40.812 INFO:journalctl@ceph.mon.a.smithi129.stdout: data_block_index_type: 0 2024-03-11T20:45:40.812 INFO:journalctl@ceph.mon.a.smithi129.stdout: index_shortening: 1 2024-03-11T20:45:40.812 INFO:journalctl@ceph.mon.a.smithi129.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T20:45:40.812 INFO:journalctl@ceph.mon.a.smithi129.stdout: checksum: 4 2024-03-11T20:45:40.812 INFO:journalctl@ceph.mon.a.smithi129.stdout: no_block_cache: 0 2024-03-11T20:45:40.812 INFO:journalctl@ceph.mon.a.smithi129.stdout: block_cache: 0x563769253090 2024-03-11T20:45:40.812 INFO:journalctl@ceph.mon.a.smithi129.stdout: block_cache_name: BinnedLRUCache 2024-03-11T20:45:40.812 INFO:journalctl@ceph.mon.a.smithi129.stdout: block_cache_options: 2024-03-11T20:45:40.812 INFO:journalctl@ceph.mon.a.smithi129.stdout: capacity : 536870912 2024-03-11T20:45:40.812 INFO:journalctl@ceph.mon.a.smithi129.stdout: num_shard_bits : 4 2024-03-11T20:45:40.812 INFO:journalctl@ceph.mon.a.smithi129.stdout: strict_capacity_limit : 0 2024-03-11T20:45:40.812 INFO:journalctl@ceph.mon.a.smithi129.stdout: high_pri_pool_ratio: 0.000 2024-03-11T20:45:40.813 INFO:journalctl@ceph.mon.a.smithi129.stdout: block_cache_compressed: (nil) 2024-03-11T20:45:40.813 INFO:journalctl@ceph.mon.a.smithi129.stdout: persistent_cache: (nil) 2024-03-11T20:45:40.813 INFO:journalctl@ceph.mon.a.smithi129.stdout: block_size: 4096 2024-03-11T20:45:40.813 INFO:journalctl@ceph.mon.a.smithi129.stdout: block_size_deviation: 10 2024-03-11T20:45:40.813 INFO:journalctl@ceph.mon.a.smithi129.stdout: block_restart_interval: 16 2024-03-11T20:45:40.813 INFO:journalctl@ceph.mon.a.smithi129.stdout: index_block_restart_interval: 1 2024-03-11T20:45:40.813 INFO:journalctl@ceph.mon.a.smithi129.stdout: metadata_block_size: 4096 2024-03-11T20:45:40.813 INFO:journalctl@ceph.mon.a.smithi129.stdout: partition_filters: 0 2024-03-11T20:45:40.813 INFO:journalctl@ceph.mon.a.smithi129.stdout: use_delta_encoding: 1 2024-03-11T20:45:40.813 INFO:journalctl@ceph.mon.a.smithi129.stdout: filter_policy: bloomfilter 2024-03-11T20:45:40.813 INFO:journalctl@ceph.mon.a.smithi129.stdout: whole_key_filtering: 1 2024-03-11T20:45:40.813 INFO:journalctl@ceph.mon.a.smithi129.stdout: verify_compression: 0 2024-03-11T20:45:40.813 INFO:journalctl@ceph.mon.a.smithi129.stdout: read_amp_bytes_per_bit: 0 2024-03-11T20:45:40.813 INFO:journalctl@ceph.mon.a.smithi129.stdout: format_version: 5 2024-03-11T20:45:40.814 INFO:journalctl@ceph.mon.a.smithi129.stdout: enable_index_compression: 1 2024-03-11T20:45:40.814 INFO:journalctl@ceph.mon.a.smithi129.stdout: block_align: 0 2024-03-11T20:45:40.814 INFO:journalctl@ceph.mon.a.smithi129.stdout: max_auto_readahead_size: 262144 2024-03-11T20:45:40.814 INFO:journalctl@ceph.mon.a.smithi129.stdout: prepopulate_block_cache: 0 2024-03-11T20:45:40.814 INFO:journalctl@ceph.mon.a.smithi129.stdout: initial_auto_readahead_size: 8192 2024-03-11T20:45:40.814 INFO:journalctl@ceph.mon.a.smithi129.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T20:45:40.814 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T20:45:40.814 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T20:45:40.814 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compression: NoCompression 2024-03-11T20:45:40.814 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T20:45:40.814 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T20:45:40.814 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T20:45:40.814 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.num_levels: 7 2024-03-11T20:45:40.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T20:45:40.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T20:45:40.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T20:45:40.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T20:45:40.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T20:45:40.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T20:45:40.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T20:45:40.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T20:45:40.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T20:45:40.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T20:45:40.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T20:45:40.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T20:45:40.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T20:45:40.815 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T20:45:40.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T20:45:40.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T20:45:40.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T20:45:40.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T20:45:40.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T20:45:40.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T20:45:40.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T20:45:40.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T20:45:40.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T20:45:40.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T20:45:40.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T20:45:40.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T20:45:40.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T20:45:40.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T20:45:40.816 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T20:45:40.817 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T20:45:40.817 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T20:45:40.817 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T20:45:40.817 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T20:45:40.817 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T20:45:40.817 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T20:45:40.817 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T20:45:40.817 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T20:45:40.817 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T20:45:40.817 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T20:45:40.817 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T20:45:40.817 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T20:45:40.817 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T20:45:40.818 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T20:45:40.818 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T20:45:40.818 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T20:45:40.818 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T20:45:40.818 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T20:45:40.818 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T20:45:40.818 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T20:45:40.818 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T20:45:40.818 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T20:45:40.818 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T20:45:40.818 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T20:45:40.818 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.table_properties_collectors: 2024-03-11T20:45:40.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.inplace_update_support: 0 2024-03-11T20:45:40.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T20:45:40.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T20:45:40.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T20:45:40.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T20:45:40.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.bloom_locality: 0 2024-03-11T20:45:40.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.max_successive_merges: 0 2024-03-11T20:45:40.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T20:45:40.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T20:45:40.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T20:45:40.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T20:45:40.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.ttl: 2592000 2024-03-11T20:45:40.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T20:45:40.819 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T20:45:40.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T20:45:40.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.enable_blob_files: false 2024-03-11T20:45:40.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.min_blob_size: 0 2024-03-11T20:45:40.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T20:45:40.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T20:45:40.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T20:45:40.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T20:45:40.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T20:45:40.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T20:45:40.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T20:45:40.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T20:45:40.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-11T20:45:40.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-11T20:45:40.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 756ca77d-5f2a-4efe-b4e4-b481bd4e7818 2024-03-11T20:45:40.820 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710189940503093, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-11T20:45:40.821 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-11T20:45:40.821 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710189940504030, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 83261, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 262, "table_properties": {"data_size": 81433, "index_size": 238, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10592, "raw_average_key_size": 47, "raw_value_size": 75445, "raw_average_value_size": 341, "num_data_blocks": 10, "num_entries": 221, "num_filter_entries": 221, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710189940, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "756ca77d-5f2a-4efe-b4e4-b481bd4e7818", "db_session_id": "G9MQU4N0FZLRMLP1RYR9", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-11T20:45:40.821 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710189940504112, "job": 1, "event": "recovery_finished"} 2024-03-11T20:45:40.821 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-11T20:45:40.821 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T20:45:40.821 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563769346000 2024-03-11T20:45:40.821 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: DB pointer 0x563769330000 2024-03-11T20:45:40.821 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T20:45:40.821 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T20:45:40.821 INFO:journalctl@ceph.mon.a.smithi129.stdout: ** DB Stats ** 2024-03-11T20:45:40.821 INFO:journalctl@ceph.mon.a.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T20:45:40.821 INFO:journalctl@ceph.mon.a.smithi129.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T20:45:40.821 INFO:journalctl@ceph.mon.a.smithi129.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T20:45:40.821 INFO:journalctl@ceph.mon.a.smithi129.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T20:45:40.822 INFO:journalctl@ceph.mon.a.smithi129.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T20:45:40.822 INFO:journalctl@ceph.mon.a.smithi129.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T20:45:40.822 INFO:journalctl@ceph.mon.a.smithi129.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T20:45:40.822 INFO:journalctl@ceph.mon.a.smithi129.stdout: 2024-03-11T20:45:40.822 INFO:journalctl@ceph.mon.a.smithi129.stdout: ** Compaction Stats [default] ** 2024-03-11T20:45:40.822 INFO:journalctl@ceph.mon.a.smithi129.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T20:45:40.822 INFO:journalctl@ceph.mon.a.smithi129.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T20:45:40.822 INFO:journalctl@ceph.mon.a.smithi129.stdout: L0 2/0 83.14 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T20:45:40.822 INFO:journalctl@ceph.mon.a.smithi129.stdout: Sum 2/0 83.14 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T20:45:40.822 INFO:journalctl@ceph.mon.a.smithi129.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T20:45:40.822 INFO:journalctl@ceph.mon.a.smithi129.stdout: 2024-03-11T20:45:40.822 INFO:journalctl@ceph.mon.a.smithi129.stdout: ** Compaction Stats [default] ** 2024-03-11T20:45:40.822 INFO:journalctl@ceph.mon.a.smithi129.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T20:45:40.822 INFO:journalctl@ceph.mon.a.smithi129.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T20:45:40.823 INFO:journalctl@ceph.mon.a.smithi129.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 126.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T20:45:40.823 INFO:journalctl@ceph.mon.a.smithi129.stdout: 2024-03-11T20:45:40.823 INFO:journalctl@ceph.mon.a.smithi129.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T20:45:40.823 INFO:journalctl@ceph.mon.a.smithi129.stdout: 2024-03-11T20:45:40.823 INFO:journalctl@ceph.mon.a.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T20:45:40.823 INFO:journalctl@ceph.mon.a.smithi129.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T20:45:40.823 INFO:journalctl@ceph.mon.a.smithi129.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T20:45:40.823 INFO:journalctl@ceph.mon.a.smithi129.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T20:45:40.823 INFO:journalctl@ceph.mon.a.smithi129.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T20:45:40.823 INFO:journalctl@ceph.mon.a.smithi129.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T20:45:40.823 INFO:journalctl@ceph.mon.a.smithi129.stdout: Cumulative compaction: 0.00 GB write, 27.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T20:45:40.823 INFO:journalctl@ceph.mon.a.smithi129.stdout: Interval compaction: 0.00 GB write, 27.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T20:45:40.823 INFO:journalctl@ceph.mon.a.smithi129.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T20:45:40.823 INFO:journalctl@ceph.mon.a.smithi129.stdout: Block cache BinnedLRUCache@0x563769253090#7 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-11T20:45:40.823 INFO:journalctl@ceph.mon.a.smithi129.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-03-11T20:45:40.824 INFO:journalctl@ceph.mon.a.smithi129.stdout: 2024-03-11T20:45:40.824 INFO:journalctl@ceph.mon.a.smithi129.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T20:45:40.824 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: starting mon.a rank 0 at public addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] at bind addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:45:40.824 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: mon.a@-1(???) e1 preinit fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:45:40.824 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: mon.a@-1(???).mds e1 new map 2024-03-11T20:45:40.824 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: mon.a@-1(???).mds e1 print_map 2024-03-11T20:45:40.824 INFO:journalctl@ceph.mon.a.smithi129.stdout: e1 2024-03-11T20:45:40.824 INFO:journalctl@ceph.mon.a.smithi129.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T20:45:40.824 INFO:journalctl@ceph.mon.a.smithi129.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T20:45:40.824 INFO:journalctl@ceph.mon.a.smithi129.stdout: legacy client fscid: -1 2024-03-11T20:45:40.824 INFO:journalctl@ceph.mon.a.smithi129.stdout: 2024-03-11T20:45:40.824 INFO:journalctl@ceph.mon.a.smithi129.stdout: No filesystems configured 2024-03-11T20:45:40.824 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T20:45:40.824 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T20:45:40.824 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T20:45:40.825 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T20:45:40.825 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T20:45:40.825 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T20:45:40.825 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: monmap e1: 1 mons at {a=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:45:40.825 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: fsmap 2024-03-11T20:45:40.825 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T20:45:40.825 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 ceph-mon[30184]: mgrmap e1: no daemons active 2024-03-11T20:45:40.825 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 bash[30146]: 4970c2d909d20e2a6c4a3d04b81de9dea01ea397a4addf67c6c955c886605dec 2024-03-11T20:45:40.825 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:40 smithi129 systemd[1]: Started Ceph mon.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T20:45:41.721 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.060+0000 7f690fd60700 1 Processor -- start 2024-03-11T20:45:41.726 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.060+0000 7f690fd60700 1 -- start start 2024-03-11T20:45:41.726 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.060+0000 7f690fd60700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908104b60 0x7f6908104f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:41.726 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.060+0000 7f690fd60700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6908105550 con 0x7f6908104b60 2024-03-11T20:45:41.726 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.061+0000 7f690dafc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908104b60 0x7f6908104f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:41.726 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.061+0000 7f690dafc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908104b60 0x7f6908104f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37056/0 (socket says 172.21.15.129:37056) 2024-03-11T20:45:41.726 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.061+0000 7f690dafc700 1 -- 172.21.15.129:0/1936454795 learned_addr learned my addr 172.21.15.129:0/1936454795 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:41.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.061+0000 7f690dafc700 1 -- 172.21.15.129:0/1936454795 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6908105d60 con 0x7f6908104b60 2024-03-11T20:45:41.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.061+0000 7f690dafc700 1 --2- 172.21.15.129:0/1936454795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908104b60 0x7f6908104f80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6904009e80 tx=0x7f690400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f0aa424cb40169 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:41.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.061+0000 7f690cafa700 1 -- 172.21.15.129:0/1936454795 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6904017070 con 0x7f6908104b60 2024-03-11T20:45:41.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.061+0000 7f690cafa700 1 -- 172.21.15.129:0/1936454795 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f690400b8c0 con 0x7f6908104b60 2024-03-11T20:45:41.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.061+0000 7f690cafa700 1 -- 172.21.15.129:0/1936454795 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6904004370 con 0x7f6908104b60 2024-03-11T20:45:41.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.062+0000 7f690fd60700 1 -- 172.21.15.129:0/1936454795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908104b60 msgr2=0x7f6908104f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:41.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.062+0000 7f690fd60700 1 --2- 172.21.15.129:0/1936454795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908104b60 0x7f6908104f80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6904009e80 tx=0x7f690400b330 comp rx=0 tx=0).stop 2024-03-11T20:45:41.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.062+0000 7f690fd60700 1 -- 172.21.15.129:0/1936454795 shutdown_connections 2024-03-11T20:45:41.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.062+0000 7f690fd60700 1 --2- 172.21.15.129:0/1936454795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908104b60 0x7f6908104f80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:41.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.062+0000 7f690fd60700 1 -- 172.21.15.129:0/1936454795 >> 172.21.15.129:0/1936454795 conn(0x7f69081000e0 msgr2=0x7f6908102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:41.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.062+0000 7f690fd60700 1 -- 172.21.15.129:0/1936454795 shutdown_connections 2024-03-11T20:45:41.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.062+0000 7f690fd60700 1 -- 172.21.15.129:0/1936454795 wait complete. 2024-03-11T20:45:41.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.063+0000 7f690fd60700 1 Processor -- start 2024-03-11T20:45:41.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.063+0000 7f690fd60700 1 -- start start 2024-03-11T20:45:41.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.063+0000 7f690fd60700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908104b60 0x7f69080792d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:41.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.063+0000 7f690fd60700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6908106bf0 con 0x7f6908104b60 2024-03-11T20:45:41.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.063+0000 7f690dafc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908104b60 0x7f69080792d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:41.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.063+0000 7f690dafc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908104b60 0x7f69080792d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37062/0 (socket says 172.21.15.129:37062) 2024-03-11T20:45:41.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.063+0000 7f690dafc700 1 -- 172.21.15.129:0/945245005 learned_addr learned my addr 172.21.15.129:0/945245005 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:41.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.063+0000 7f690dafc700 1 -- 172.21.15.129:0/945245005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f690819c780 con 0x7f6908104b60 2024-03-11T20:45:41.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.063+0000 7f690dafc700 1 --2- 172.21.15.129:0/945245005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908104b60 0x7f69080792d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f69040072a0 tx=0x7f6904003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:41.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.064+0000 7f68feffd700 1 -- 172.21.15.129:0/945245005 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6904021020 con 0x7f6908104b60 2024-03-11T20:45:41.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.064+0000 7f68feffd700 1 -- 172.21.15.129:0/945245005 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f690400bde0 con 0x7f6908104b60 2024-03-11T20:45:41.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.064+0000 7f68feffd700 1 -- 172.21.15.129:0/945245005 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6904017040 con 0x7f6908104b60 2024-03-11T20:45:41.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.064+0000 7f690fd60700 1 -- 172.21.15.129:0/945245005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f690819c950 con 0x7f6908104b60 2024-03-11T20:45:41.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.064+0000 7f690fd60700 1 -- 172.21.15.129:0/945245005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f690819cd70 con 0x7f6908104b60 2024-03-11T20:45:41.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.064+0000 7f68feffd700 1 -- 172.21.15.129:0/945245005 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6904024020 con 0x7f6908104b60 2024-03-11T20:45:41.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.064+0000 7f68feffd700 1 -- 172.21.15.129:0/945245005 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f690400f050 con 0x7f6908104b60 2024-03-11T20:45:41.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.065+0000 7f690fd60700 1 -- 172.21.15.129:0/945245005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6908063410 con 0x7f6908104b60 2024-03-11T20:45:41.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.067+0000 7f68feffd700 1 -- 172.21.15.129:0/945245005 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f690402b020 con 0x7f6908104b60 2024-03-11T20:45:41.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.133+0000 7f690fd60700 1 -- 172.21.15.129:0/945245005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f690819d2b0 con 0x7f6908104b60 2024-03-11T20:45:41.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.135+0000 7f68feffd700 1 -- 172.21.15.129:0/945245005 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f690400b860 con 0x7f6908104b60 2024-03-11T20:45:41.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.136+0000 7f690fd60700 1 -- 172.21.15.129:0/945245005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908104b60 msgr2=0x7f69080792d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:41.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.136+0000 7f690fd60700 1 --2- 172.21.15.129:0/945245005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908104b60 0x7f69080792d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f69040072a0 tx=0x7f6904003d00 comp rx=0 tx=0).stop 2024-03-11T20:45:41.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.136+0000 7f690fd60700 1 -- 172.21.15.129:0/945245005 shutdown_connections 2024-03-11T20:45:41.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.136+0000 7f690fd60700 1 --2- 172.21.15.129:0/945245005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6908104b60 0x7f69080792d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:41.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.136+0000 7f690fd60700 1 -- 172.21.15.129:0/945245005 >> 172.21.15.129:0/945245005 conn(0x7f69081000e0 msgr2=0x7f6908100c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:41.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.136+0000 7f690fd60700 1 -- 172.21.15.129:0/945245005 shutdown_connections 2024-03-11T20:45:41.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:41.136+0000 7f690fd60700 1 -- 172.21.15.129:0/945245005 wait complete. 2024-03-11T20:45:41.729 INFO:teuthology.orchestra.run.smithi129.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-11T20:45:41.744 INFO:teuthology.orchestra.run.smithi129.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T20:45:41.744 INFO:teuthology.orchestra.run.smithi129.stdout:Creating mgr... 2024-03-11T20:45:41.744 INFO:teuthology.orchestra.run.smithi129.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-11T20:45:41.745 INFO:teuthology.orchestra.run.smithi129.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-11T20:45:41.991 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mgr.y 2024-03-11T20:45:41.992 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Failed to reset failed state of unit ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-11T20:45:42.134 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715.target.wants/ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@.service. 2024-03-11T20:45:42.362 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/945245005' entity='client.admin' 2024-03-11T20:45:42.362 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:42 smithi129 systemd[1]: Starting Ceph mgr.y for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T20:45:42.615 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:42 smithi129 podman[30423]: 2024-03-11 20:45:42.27085526 +0000 UTC m=+0.017137327 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T20:45:42.615 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:42 smithi129 podman[30423]: 2024-03-11 20:45:42.368887013 +0000 UTC m=+0.115169070 container create a9b2778718397dc47690fde33a1b9ff1c56364ffcb1100eaa93365a7a628c018 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T20:45:42.615 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:42 smithi129 podman[30423]: 2024-03-11 20:45:42.62119282 +0000 UTC m=+0.367474873 container init a9b2778718397dc47690fde33a1b9ff1c56364ffcb1100eaa93365a7a628c018 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-11T20:45:42.738 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T20:45:42.739 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout disabled 2024-03-11T20:45:42.747 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T20:45:42.747 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout inactive 2024-03-11T20:45:42.747 INFO:teuthology.orchestra.run.smithi129.stdout:firewalld.service is not enabled 2024-03-11T20:45:42.747 INFO:teuthology.orchestra.run.smithi129.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T20:45:42.757 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T20:45:42.757 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout disabled 2024-03-11T20:45:42.766 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T20:45:42.766 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout inactive 2024-03-11T20:45:42.766 INFO:teuthology.orchestra.run.smithi129.stdout:firewalld.service is not enabled 2024-03-11T20:45:42.766 INFO:teuthology.orchestra.run.smithi129.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-11T20:45:42.766 INFO:teuthology.orchestra.run.smithi129.stdout:Waiting for mgr to start... 2024-03-11T20:45:42.766 INFO:teuthology.orchestra.run.smithi129.stdout:Waiting for mgr... 2024-03-11T20:45:42.883 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:42 smithi129 podman[30423]: 2024-03-11 20:45:42.626364506 +0000 UTC m=+0.372646562 container start a9b2778718397dc47690fde33a1b9ff1c56364ffcb1100eaa93365a7a628c018 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T20:45:42.883 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:42 smithi129 bash[30423]: a9b2778718397dc47690fde33a1b9ff1c56364ffcb1100eaa93365a7a628c018 2024-03-11T20:45:42.883 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:42 smithi129 systemd[1]: Started Ceph mgr.y for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T20:45:43.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2199726137' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T20:45:43.477 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:43 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:43.193+0000 7f65bc63d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T20:45:43.477 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:43 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:43.290+0000 7f65bc63d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T20:45:43.869 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-03-11T20:45:43.869 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout { 2024-03-11T20:45:43.869 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "fsid": "27bbebfa-dfe8-11ee-95c8-87774f69a715", 2024-03-11T20:45:43.870 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T20:45:43.870 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T20:45:43.870 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T20:45:43.870 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T20:45:43.870 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:43.870 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T20:45:43.870 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T20:45:43.870 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 0 2024-03-11T20:45:43.870 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:45:43.870 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T20:45:43.870 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T20:45:43.870 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:45:43.870 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-11T20:45:43.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T20:45:43.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:45:43.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T20:45:43.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T20:45:43.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:43.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T20:45:43.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:45:43.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T20:45:43.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T20:45:43.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T20:45:43.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T20:45:43.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T20:45:43.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T20:45:43.871 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:43.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T20:45:43.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T20:45:43.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T20:45:43.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T20:45:43.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T20:45:43.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T20:45:43.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T20:45:43.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T20:45:43.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T20:45:43.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:43.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T20:45:43.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:45:43.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T20:45:43.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T20:45:43.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:43.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T20:45:43.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T20:45:43.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T20:45:43.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T20:45:43.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T20:45:43.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T20:45:43.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T20:45:43.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:45:43.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T20:45:43.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:43.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T20:45:43.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:45:43.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T20:45:35.171345+0000", 2024-03-11T20:45:43.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T20:45:43.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:43.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T20:45:43.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout } 2024-03-11T20:45:43.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.235+0000 7fc880489700 1 Processor -- start 2024-03-11T20:45:43.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.235+0000 7fc880489700 1 -- start start 2024-03-11T20:45:43.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.235+0000 7fc880489700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc878104b60 0x7fc878104f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:43.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.235+0000 7fc880489700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc878105550 con 0x7fc878104b60 2024-03-11T20:45:43.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.235+0000 7fc87e225700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc878104b60 0x7fc878104f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:43.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.235+0000 7fc87e225700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc878104b60 0x7fc878104f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37090/0 (socket says 172.21.15.129:37090) 2024-03-11T20:45:43.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.235+0000 7fc87e225700 1 -- 172.21.15.129:0/3118820686 learned_addr learned my addr 172.21.15.129:0/3118820686 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:43.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.236+0000 7fc87e225700 1 -- 172.21.15.129:0/3118820686 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc878105d60 con 0x7fc878104b60 2024-03-11T20:45:43.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.236+0000 7fc87e225700 1 --2- 172.21.15.129:0/3118820686 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc878104b60 0x7fc878104f80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc8740098a0 tx=0x7fc874009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff8dee4b80323d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:43.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.236+0000 7fc87d223700 1 -- 172.21.15.129:0/3118820686 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc874004160 con 0x7fc878104b60 2024-03-11T20:45:43.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.236+0000 7fc87d223700 1 -- 172.21.15.129:0/3118820686 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fc874004300 con 0x7fc878104b60 2024-03-11T20:45:43.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.236+0000 7fc87d223700 1 -- 172.21.15.129:0/3118820686 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc874011940 con 0x7fc878104b60 2024-03-11T20:45:43.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.237+0000 7fc880489700 1 -- 172.21.15.129:0/3118820686 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc878104b60 msgr2=0x7fc878104f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:43.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.237+0000 7fc880489700 1 --2- 172.21.15.129:0/3118820686 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc878104b60 0x7fc878104f80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc8740098a0 tx=0x7fc874009bb0 comp rx=0 tx=0).stop 2024-03-11T20:45:43.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.237+0000 7fc880489700 1 -- 172.21.15.129:0/3118820686 shutdown_connections 2024-03-11T20:45:43.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.237+0000 7fc880489700 1 --2- 172.21.15.129:0/3118820686 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc878104b60 0x7fc878104f80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:43.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.237+0000 7fc880489700 1 -- 172.21.15.129:0/3118820686 >> 172.21.15.129:0/3118820686 conn(0x7fc8781000e0 msgr2=0x7fc878102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:43.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.237+0000 7fc880489700 1 -- 172.21.15.129:0/3118820686 shutdown_connections 2024-03-11T20:45:43.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.237+0000 7fc880489700 1 -- 172.21.15.129:0/3118820686 wait complete. 2024-03-11T20:45:43.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.238+0000 7fc880489700 1 Processor -- start 2024-03-11T20:45:43.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.238+0000 7fc880489700 1 -- start start 2024-03-11T20:45:43.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.238+0000 7fc880489700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc878104b60 0x7fc878199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:43.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.238+0000 7fc880489700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc878106bf0 con 0x7fc878104b60 2024-03-11T20:45:43.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.238+0000 7fc87e225700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc878104b60 0x7fc878199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:43.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.238+0000 7fc87e225700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc878104b60 0x7fc878199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37094/0 (socket says 172.21.15.129:37094) 2024-03-11T20:45:43.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.238+0000 7fc87e225700 1 -- 172.21.15.129:0/2199726137 learned_addr learned my addr 172.21.15.129:0/2199726137 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:43.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.238+0000 7fc87e225700 1 -- 172.21.15.129:0/2199726137 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc878199ff0 con 0x7fc878104b60 2024-03-11T20:45:43.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.238+0000 7fc87e225700 1 --2- 172.21.15.129:0/2199726137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc878104b60 0x7fc878199ab0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc874006ba0 tx=0x7fc8740119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:43.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.238+0000 7fc86f7fe700 1 -- 172.21.15.129:0/2199726137 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc874003930 con 0x7fc878104b60 2024-03-11T20:45:43.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.238+0000 7fc880489700 1 -- 172.21.15.129:0/2199726137 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87819a280 con 0x7fc878104b60 2024-03-11T20:45:43.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.238+0000 7fc86f7fe700 1 -- 172.21.15.129:0/2199726137 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fc874003ad0 con 0x7fc878104b60 2024-03-11T20:45:43.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.238+0000 7fc86f7fe700 1 -- 172.21.15.129:0/2199726137 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc874019750 con 0x7fc878104b60 2024-03-11T20:45:43.877 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.238+0000 7fc880489700 1 -- 172.21.15.129:0/2199726137 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87819cf70 con 0x7fc878104b60 2024-03-11T20:45:43.877 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.239+0000 7fc86f7fe700 1 -- 172.21.15.129:0/2199726137 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc87402d440 con 0x7fc878104b60 2024-03-11T20:45:43.877 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.239+0000 7fc86f7fe700 1 -- 172.21.15.129:0/2199726137 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc8740247c0 con 0x7fc878104b60 2024-03-11T20:45:43.877 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.239+0000 7fc880489700 1 -- 172.21.15.129:0/2199726137 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc878063410 con 0x7fc878104b60 2024-03-11T20:45:43.877 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.242+0000 7fc86f7fe700 1 -- 172.21.15.129:0/2199726137 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc874043dd0 con 0x7fc878104b60 2024-03-11T20:45:43.877 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.314+0000 7fc880489700 1 -- 172.21.15.129:0/2199726137 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc87819d2c0 con 0x7fc878104b60 2024-03-11T20:45:43.877 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.315+0000 7fc86f7fe700 1 -- 172.21.15.129:0/2199726137 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fc874032020 con 0x7fc878104b60 2024-03-11T20:45:43.877 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.316+0000 7fc880489700 1 -- 172.21.15.129:0/2199726137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc878104b60 msgr2=0x7fc878199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:43.877 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.316+0000 7fc880489700 1 --2- 172.21.15.129:0/2199726137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc878104b60 0x7fc878199ab0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc874006ba0 tx=0x7fc8740119c0 comp rx=0 tx=0).stop 2024-03-11T20:45:43.877 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.316+0000 7fc880489700 1 -- 172.21.15.129:0/2199726137 shutdown_connections 2024-03-11T20:45:43.877 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.316+0000 7fc880489700 1 --2- 172.21.15.129:0/2199726137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc878104b60 0x7fc878199ab0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:43.877 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.316+0000 7fc880489700 1 -- 172.21.15.129:0/2199726137 >> 172.21.15.129:0/2199726137 conn(0x7fc8781000e0 msgr2=0x7fc878100cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:43.877 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.316+0000 7fc880489700 1 -- 172.21.15.129:0/2199726137 shutdown_connections 2024-03-11T20:45:43.877 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:43.316+0000 7fc880489700 1 -- 172.21.15.129:0/2199726137 wait complete. 2024-03-11T20:45:43.877 INFO:teuthology.orchestra.run.smithi129.stdout:mgr not available, waiting (1/15)... 2024-03-11T20:45:44.227 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:43 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:43.950+0000 7f65bc63d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T20:45:44.227 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:44 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:44.042+0000 7f65bc63d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T20:45:44.227 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:44 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:44.183+0000 7f65bc63d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T20:45:44.612 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:44 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:44.277+0000 7f65bc63d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T20:45:44.612 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:44 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:44.358+0000 7f65bc63d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T20:45:44.612 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:44 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:44.617+0000 7f65bc63d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T20:45:44.896 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:44 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:44.726+0000 7f65bc63d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T20:45:44.896 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:44 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:44.809+0000 7f65bc63d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T20:45:44.896 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:44 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:44.901+0000 7f65bc63d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T20:45:45.227 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:45 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:45.077+0000 7f65bc63d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T20:45:46.282 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:46.043+0000 7f65bc63d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T20:45:46.282 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:46.184+0000 7f65bc63d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T20:45:46.533 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3710613425' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T20:45:46.823 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:46.539+0000 7f65bc63d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T20:45:46.823 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:46.633+0000 7f65bc63d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T20:45:46.823 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:46.828+0000 7f65bc63d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:45:47.024 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-03-11T20:45:47.024 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout { 2024-03-11T20:45:47.024 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "fsid": "27bbebfa-dfe8-11ee-95c8-87774f69a715", 2024-03-11T20:45:47.024 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T20:45:47.024 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T20:45:47.024 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T20:45:47.024 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T20:45:47.024 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:47.024 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T20:45:47.024 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T20:45:47.025 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 0 2024-03-11T20:45:47.025 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:45:47.025 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T20:45:47.025 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T20:45:47.025 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:45:47.025 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-11T20:45:47.025 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T20:45:47.025 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:45:47.025 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T20:45:47.025 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T20:45:47.025 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:47.025 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T20:45:47.025 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:45:47.025 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T20:45:47.025 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T20:45:47.026 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T20:45:47.033 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T20:45:47.033 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T20:45:47.033 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T20:45:47.033 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:47.033 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T20:45:47.033 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T20:45:47.033 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T20:45:47.034 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T20:45:47.034 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T20:45:47.034 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T20:45:47.034 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T20:45:47.034 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T20:45:47.034 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T20:45:47.034 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:47.034 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T20:45:47.034 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:45:47.034 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T20:45:47.034 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T20:45:47.034 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:47.034 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T20:45:47.035 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T20:45:47.035 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T20:45:47.035 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T20:45:47.035 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T20:45:47.035 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T20:45:47.035 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T20:45:47.035 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:45:47.035 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T20:45:47.035 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:47.035 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T20:45:47.035 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:45:47.035 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T20:45:35.171345+0000", 2024-03-11T20:45:47.035 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T20:45:47.035 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:47.035 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T20:45:47.036 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout } 2024-03-11T20:45:47.036 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.315+0000 7f61f960e700 1 Processor -- start 2024-03-11T20:45:47.036 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.315+0000 7f61f960e700 1 -- start start 2024-03-11T20:45:47.036 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.315+0000 7f61f960e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4106780 0x7f61f4108b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:47.036 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.315+0000 7f61f960e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f4079650 con 0x7f61f4106780 2024-03-11T20:45:47.036 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.315+0000 7f61f2ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4106780 0x7f61f4108b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:47.036 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.316+0000 7f61f2ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4106780 0x7f61f4108b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38866/0 (socket says 172.21.15.129:38866) 2024-03-11T20:45:47.036 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.316+0000 7f61f2ffd700 1 -- 172.21.15.129:0/4115255831 learned_addr learned my addr 172.21.15.129:0/4115255831 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:47.036 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.316+0000 7f61f2ffd700 1 -- 172.21.15.129:0/4115255831 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f41090b0 con 0x7f61f4106780 2024-03-11T20:45:47.036 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.316+0000 7f61f2ffd700 1 --2- 172.21.15.129:0/4115255831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4106780 0x7f61f4108b70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f61ec009e80 tx=0x7f61ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53fd65ad72488967 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:47.036 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.316+0000 7f61f1ffb700 1 -- 172.21.15.129:0/4115255831 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f61ec017070 con 0x7f61f4106780 2024-03-11T20:45:47.036 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.316+0000 7f61f1ffb700 1 -- 172.21.15.129:0/4115255831 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f61ec00b8c0 con 0x7f61f4106780 2024-03-11T20:45:47.036 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.316+0000 7f61f1ffb700 1 -- 172.21.15.129:0/4115255831 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f61ec004370 con 0x7f61f4106780 2024-03-11T20:45:47.036 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.317+0000 7f61f960e700 1 -- 172.21.15.129:0/4115255831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4106780 msgr2=0x7f61f4108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:47.037 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.317+0000 7f61f960e700 1 --2- 172.21.15.129:0/4115255831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4106780 0x7f61f4108b70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f61ec009e80 tx=0x7f61ec00b330 comp rx=0 tx=0).stop 2024-03-11T20:45:47.037 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.317+0000 7f61f960e700 1 -- 172.21.15.129:0/4115255831 shutdown_connections 2024-03-11T20:45:47.037 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.317+0000 7f61f960e700 1 --2- 172.21.15.129:0/4115255831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4106780 0x7f61f4108b70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:47.037 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.317+0000 7f61f960e700 1 -- 172.21.15.129:0/4115255831 >> 172.21.15.129:0/4115255831 conn(0x7f61f41000b0 msgr2=0x7f61f4102510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:47.037 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.317+0000 7f61f960e700 1 -- 172.21.15.129:0/4115255831 shutdown_connections 2024-03-11T20:45:47.037 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.317+0000 7f61f960e700 1 -- 172.21.15.129:0/4115255831 wait complete. 2024-03-11T20:45:47.037 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.317+0000 7f61f960e700 1 Processor -- start 2024-03-11T20:45:47.037 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.318+0000 7f61f960e700 1 -- start start 2024-03-11T20:45:47.037 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.318+0000 7f61f960e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4106780 0x7f61f4106000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:47.037 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.318+0000 7f61f960e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f4109e10 con 0x7f61f4106780 2024-03-11T20:45:47.037 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.318+0000 7f61f2ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4106780 0x7f61f4106000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:47.037 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.318+0000 7f61f2ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4106780 0x7f61f4106000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38880/0 (socket says 172.21.15.129:38880) 2024-03-11T20:45:47.037 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.318+0000 7f61f2ffd700 1 -- 172.21.15.129:0/3710613425 learned_addr learned my addr 172.21.15.129:0/3710613425 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:47.037 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.318+0000 7f61f2ffd700 1 -- 172.21.15.129:0/3710613425 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f4106540 con 0x7f61f4106780 2024-03-11T20:45:47.038 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.318+0000 7f61f2ffd700 1 --2- 172.21.15.129:0/3710613425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4106780 0x7f61f4106000 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0072a0 tx=0x7f61ec003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:47.038 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.319+0000 7f61ebfff700 1 -- 172.21.15.129:0/3710613425 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f61ec021020 con 0x7f61f4106780 2024-03-11T20:45:47.038 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.319+0000 7f61ebfff700 1 -- 172.21.15.129:0/3710613425 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f61ec00bde0 con 0x7f61f4106780 2024-03-11T20:45:47.038 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.319+0000 7f61ebfff700 1 -- 172.21.15.129:0/3710613425 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f61ec017040 con 0x7f61f4106780 2024-03-11T20:45:47.038 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.319+0000 7f61f960e700 1 -- 172.21.15.129:0/3710613425 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f4104730 con 0x7f61f4106780 2024-03-11T20:45:47.038 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.319+0000 7f61f960e700 1 -- 172.21.15.129:0/3710613425 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f4104c70 con 0x7f61f4106780 2024-03-11T20:45:47.038 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.319+0000 7f61ebfff700 1 -- 172.21.15.129:0/3710613425 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f61ec024020 con 0x7f61f4106780 2024-03-11T20:45:47.038 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.319+0000 7f61ebfff700 1 -- 172.21.15.129:0/3710613425 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f61ec00f050 con 0x7f61f4106780 2024-03-11T20:45:47.038 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.320+0000 7f61f960e700 1 -- 172.21.15.129:0/3710613425 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f4063410 con 0x7f61f4106780 2024-03-11T20:45:47.038 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.322+0000 7f61ebfff700 1 -- 172.21.15.129:0/3710613425 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f61ec02b020 con 0x7f61f4106780 2024-03-11T20:45:47.038 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.406+0000 7f61f960e700 1 -- 172.21.15.129:0/3710613425 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f61f4104e40 con 0x7f61f4106780 2024-03-11T20:45:47.038 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.407+0000 7f61ebfff700 1 -- 172.21.15.129:0/3710613425 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f61ec00b910 con 0x7f61f4106780 2024-03-11T20:45:47.038 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.408+0000 7f61f960e700 1 -- 172.21.15.129:0/3710613425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4106780 msgr2=0x7f61f4106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:47.039 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.408+0000 7f61f960e700 1 --2- 172.21.15.129:0/3710613425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4106780 0x7f61f4106000 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0072a0 tx=0x7f61ec003d00 comp rx=0 tx=0).stop 2024-03-11T20:45:47.039 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.408+0000 7f61f960e700 1 -- 172.21.15.129:0/3710613425 shutdown_connections 2024-03-11T20:45:47.039 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.408+0000 7f61f960e700 1 --2- 172.21.15.129:0/3710613425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61f4106780 0x7f61f4106000 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:47.039 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.408+0000 7f61f960e700 1 -- 172.21.15.129:0/3710613425 >> 172.21.15.129:0/3710613425 conn(0x7f61f41000b0 msgr2=0x7f61f4100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:47.039 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.408+0000 7f61f960e700 1 -- 172.21.15.129:0/3710613425 shutdown_connections 2024-03-11T20:45:47.039 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:46.408+0000 7f61f960e700 1 -- 172.21.15.129:0/3710613425 wait complete. 2024-03-11T20:45:47.039 INFO:teuthology.orchestra.run.smithi129.stdout:mgr not available, waiting (2/15)... 2024-03-11T20:45:47.227 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:47 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:47.013+0000 7f65bc63d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T20:45:47.556 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:47 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:47.299+0000 7f65bc63d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:45:47.556 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:47 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:47.561+0000 7f65bc63d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T20:45:47.821 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:47 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:47.797+0000 7f65bc63d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T20:45:48.977 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:48 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:48.622+0000 7f65bc63d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T20:45:49.455 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:49 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:49.217+0000 7f65bc63d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T20:45:49.456 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:49 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:49.332+0000 7f65bc63d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T20:45:49.727 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:49 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:49.558+0000 7f65bc63d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T20:45:49.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:49 smithi129 ceph-mon[30184]: Activating manager daemon y 2024-03-11T20:45:49.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:49 smithi129 ceph-mon[30184]: mgrmap e2: y(active, starting, since 0.00294425s) 2024-03-11T20:45:49.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:49 smithi129 ceph-mon[30184]: from='mgr.14100 172.21.15.129:0/529823039' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T20:45:49.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:49 smithi129 ceph-mon[30184]: from='mgr.14100 172.21.15.129:0/529823039' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T20:45:49.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:49 smithi129 ceph-mon[30184]: from='mgr.14100 172.21.15.129:0/529823039' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T20:45:49.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:49 smithi129 ceph-mon[30184]: from='mgr.14100 172.21.15.129:0/529823039' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:45:49.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:49 smithi129 ceph-mon[30184]: from='mgr.14100 172.21.15.129:0/529823039' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T20:45:49.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:49 smithi129 ceph-mon[30184]: Manager daemon y is now available 2024-03-11T20:45:49.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:49 smithi129 ceph-mon[30184]: from='mgr.14100 172.21.15.129:0/529823039' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T20:45:49.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:49 smithi129 ceph-mon[30184]: from='mgr.14100 172.21.15.129:0/529823039' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T20:45:49.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:49 smithi129 ceph-mon[30184]: from='mgr.14100 172.21.15.129:0/529823039' entity='mgr.y' 2024-03-11T20:45:49.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:49 smithi129 ceph-mon[30184]: from='mgr.14100 172.21.15.129:0/529823039' entity='mgr.y' 2024-03-11T20:45:49.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:49 smithi129 ceph-mon[30184]: from='mgr.14100 172.21.15.129:0/529823039' entity='mgr.y' 2024-03-11T20:45:50.329 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-03-11T20:45:50.339 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout { 2024-03-11T20:45:50.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "fsid": "27bbebfa-dfe8-11ee-95c8-87774f69a715", 2024-03-11T20:45:50.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T20:45:50.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T20:45:50.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T20:45:50.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T20:45:50.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:50.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T20:45:50.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T20:45:50.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 0 2024-03-11T20:45:50.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:45:50.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T20:45:50.340 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T20:45:50.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:45:50.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-11T20:45:50.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T20:45:50.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:45:50.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T20:45:50.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T20:45:50.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:50.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T20:45:50.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:45:50.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T20:45:50.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T20:45:50.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T20:45:50.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T20:45:50.341 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T20:45:50.342 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T20:45:50.342 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:50.342 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T20:45:50.342 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T20:45:50.342 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T20:45:50.342 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T20:45:50.342 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T20:45:50.342 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T20:45:50.342 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T20:45:50.342 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T20:45:50.342 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T20:45:50.342 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:50.342 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T20:45:50.343 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:45:50.343 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T20:45:50.343 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T20:45:50.343 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:50.343 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T20:45:50.343 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T20:45:50.343 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T20:45:50.343 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T20:45:50.343 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T20:45:50.343 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T20:45:50.343 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T20:45:50.343 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:45:50.343 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T20:45:50.343 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:50.344 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T20:45:50.344 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:45:50.344 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T20:45:35.171345+0000", 2024-03-11T20:45:50.344 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T20:45:50.344 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:50.344 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T20:45:50.344 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout } 2024-03-11T20:45:50.344 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.565+0000 7f151437d700 1 Processor -- start 2024-03-11T20:45:50.344 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.565+0000 7f151437d700 1 -- start start 2024-03-11T20:45:50.344 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.566+0000 7f151437d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f150c103ef0 0x7f150c104310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:50.344 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.566+0000 7f151437d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150c1048e0 con 0x7f150c103ef0 2024-03-11T20:45:50.344 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.566+0000 7f1512119700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f150c103ef0 0x7f150c104310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:50.344 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.566+0000 7f1512119700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f150c103ef0 0x7f150c104310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38896/0 (socket says 172.21.15.129:38896) 2024-03-11T20:45:50.344 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.566+0000 7f1512119700 1 -- 172.21.15.129:0/600125378 learned_addr learned my addr 172.21.15.129:0/600125378 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:50.344 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.566+0000 7f1512119700 1 -- 172.21.15.129:0/600125378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c104a60 con 0x7f150c103ef0 2024-03-11T20:45:50.345 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.567+0000 7f1512119700 1 --2- 172.21.15.129:0/600125378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f150c103ef0 0x7f150c104310 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0098a0 tx=0x7f14fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a75cbc52d7a6fd server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:50.345 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.567+0000 7f1511117700 1 -- 172.21.15.129:0/600125378 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14fc004160 con 0x7f150c103ef0 2024-03-11T20:45:50.345 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.567+0000 7f1511117700 1 -- 172.21.15.129:0/600125378 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f14fc004300 con 0x7f150c103ef0 2024-03-11T20:45:50.345 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.567+0000 7f1511117700 1 -- 172.21.15.129:0/600125378 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14fc011940 con 0x7f150c103ef0 2024-03-11T20:45:50.345 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.567+0000 7f151437d700 1 -- 172.21.15.129:0/600125378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f150c103ef0 msgr2=0x7f150c104310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:50.345 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.567+0000 7f151437d700 1 --2- 172.21.15.129:0/600125378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f150c103ef0 0x7f150c104310 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0098a0 tx=0x7f14fc009bb0 comp rx=0 tx=0).stop 2024-03-11T20:45:50.345 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.567+0000 7f151437d700 1 -- 172.21.15.129:0/600125378 shutdown_connections 2024-03-11T20:45:50.345 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.567+0000 7f151437d700 1 --2- 172.21.15.129:0/600125378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f150c103ef0 0x7f150c104310 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:50.345 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.567+0000 7f151437d700 1 -- 172.21.15.129:0/600125378 >> 172.21.15.129:0/600125378 conn(0x7f150c0ff850 msgr2=0x7f150c101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:50.345 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.567+0000 7f151437d700 1 -- 172.21.15.129:0/600125378 shutdown_connections 2024-03-11T20:45:50.345 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.567+0000 7f151437d700 1 -- 172.21.15.129:0/600125378 wait complete. 2024-03-11T20:45:50.345 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.568+0000 7f151437d700 1 Processor -- start 2024-03-11T20:45:50.345 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.568+0000 7f151437d700 1 -- start start 2024-03-11T20:45:50.345 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.568+0000 7f151437d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f150c103ef0 0x7f150c199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:50.346 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.568+0000 7f151437d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150c107c30 con 0x7f150c103ef0 2024-03-11T20:45:50.346 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.568+0000 7f1512119700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f150c103ef0 0x7f150c199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:50.346 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.568+0000 7f1512119700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f150c103ef0 0x7f150c199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38908/0 (socket says 172.21.15.129:38908) 2024-03-11T20:45:50.346 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.568+0000 7f1512119700 1 -- 172.21.15.129:0/144899687 learned_addr learned my addr 172.21.15.129:0/144899687 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:50.346 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.568+0000 7f1512119700 1 -- 172.21.15.129:0/144899687 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c199fa0 con 0x7f150c103ef0 2024-03-11T20:45:50.346 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.569+0000 7f1512119700 1 --2- 172.21.15.129:0/144899687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f150c103ef0 0x7f150c199a60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f14fc011910 tx=0x7f14fc0119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:50.346 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.569+0000 7f15037fe700 1 -- 172.21.15.129:0/144899687 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14fc003b20 con 0x7f150c103ef0 2024-03-11T20:45:50.346 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.569+0000 7f15037fe700 1 -- 172.21.15.129:0/144899687 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f14fc01a890 con 0x7f150c103ef0 2024-03-11T20:45:50.346 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.569+0000 7f15037fe700 1 -- 172.21.15.129:0/144899687 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14fc0197f0 con 0x7f150c103ef0 2024-03-11T20:45:50.346 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.569+0000 7f151437d700 1 -- 172.21.15.129:0/144899687 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150c19a230 con 0x7f150c103ef0 2024-03-11T20:45:50.346 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.569+0000 7f151437d700 1 -- 172.21.15.129:0/144899687 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f150c19cf20 con 0x7f150c103ef0 2024-03-11T20:45:50.346 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.569+0000 7f15037fe700 1 -- 172.21.15.129:0/144899687 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f14fc022020 con 0x7f150c103ef0 2024-03-11T20:45:50.346 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.569+0000 7f15037fe700 1 -- 172.21.15.129:0/144899687 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f14fc02c3a0 con 0x7f150c103ef0 2024-03-11T20:45:50.347 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.570+0000 7f151437d700 1 -- 172.21.15.129:0/144899687 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f150c063410 con 0x7f150c103ef0 2024-03-11T20:45:50.347 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.572+0000 7f15037fe700 1 -- 172.21.15.129:0/144899687 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f14fc01aa40 con 0x7f150c103ef0 2024-03-11T20:45:50.347 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.574+0000 7f15037fe700 1 -- 172.21.15.129:0/144899687 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mgrmap(e 2) v1 ==== 45862+0+0 (secure 0 0 0) 0x7f14fc01a430 con 0x7f150c103ef0 2024-03-11T20:45:50.347 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.650+0000 7f151437d700 1 -- 172.21.15.129:0/144899687 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f150c19d270 con 0x7f150c103ef0 2024-03-11T20:45:50.347 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.651+0000 7f15037fe700 1 -- 172.21.15.129:0/144899687 <== mon.0 v2:172.21.15.129:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f14fc028650 con 0x7f150c103ef0 2024-03-11T20:45:50.347 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.651+0000 7f151437d700 1 -- 172.21.15.129:0/144899687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f150c103ef0 msgr2=0x7f150c199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:50.347 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.651+0000 7f151437d700 1 --2- 172.21.15.129:0/144899687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f150c103ef0 0x7f150c199a60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f14fc011910 tx=0x7f14fc0119f0 comp rx=0 tx=0).stop 2024-03-11T20:45:50.347 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.651+0000 7f151437d700 1 -- 172.21.15.129:0/144899687 shutdown_connections 2024-03-11T20:45:50.347 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.651+0000 7f151437d700 1 --2- 172.21.15.129:0/144899687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f150c103ef0 0x7f150c199a60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:50.347 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.651+0000 7f151437d700 1 -- 172.21.15.129:0/144899687 >> 172.21.15.129:0/144899687 conn(0x7f150c0ff850 msgr2=0x7f150c100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:50.347 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.652+0000 7f151437d700 1 -- 172.21.15.129:0/144899687 shutdown_connections 2024-03-11T20:45:50.347 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:49.652+0000 7f151437d700 1 -- 172.21.15.129:0/144899687 wait complete. 2024-03-11T20:45:50.347 INFO:teuthology.orchestra.run.smithi129.stdout:mgr not available, waiting (3/15)... 2024-03-11T20:45:50.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/144899687' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T20:45:50.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:50 smithi129 ceph-mon[30184]: mgrmap e3: y(active, since 1.00703s) 2024-03-11T20:45:52.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:52 smithi129 ceph-mon[30184]: mgrmap e4: y(active, since 2s) 2024-03-11T20:45:53.757 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:53 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/917640208' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T20:45:53.762 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-03-11T20:45:53.762 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout { 2024-03-11T20:45:53.762 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "fsid": "27bbebfa-dfe8-11ee-95c8-87774f69a715", 2024-03-11T20:45:53.762 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T20:45:53.762 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T20:45:53.762 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T20:45:53.762 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T20:45:53.762 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:53.762 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T20:45:53.762 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T20:45:53.763 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 0 2024-03-11T20:45:53.763 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:45:53.763 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T20:45:53.763 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T20:45:53.763 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:45:53.763 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-11T20:45:53.763 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T20:45:53.763 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:45:53.763 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T20:45:53.763 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T20:45:53.763 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:53.763 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T20:45:53.763 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:45:53.763 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T20:45:53.763 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T20:45:53.772 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T20:45:53.772 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T20:45:53.772 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T20:45:53.772 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T20:45:53.772 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:53.772 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T20:45:53.772 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T20:45:53.772 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T20:45:53.772 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T20:45:53.773 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T20:45:53.773 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T20:45:53.773 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T20:45:53.773 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T20:45:53.773 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T20:45:53.773 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:53.773 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T20:45:53.773 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:45:53.773 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T20:45:53.773 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T20:45:53.773 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:53.773 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T20:45:53.773 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T20:45:53.773 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T20:45:53.773 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T20:45:53.774 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T20:45:53.774 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T20:45:53.774 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T20:45:53.774 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ], 2024-03-11T20:45:53.774 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T20:45:53.774 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:53.774 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T20:45:53.774 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T20:45:53.774 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T20:45:35.171345+0000", 2024-03-11T20:45:53.774 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T20:45:53.774 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout }, 2024-03-11T20:45:53.774 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T20:45:53.774 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout } 2024-03-11T20:45:53.774 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.863+0000 7f48e5fd9700 1 Processor -- start 2024-03-11T20:45:53.775 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.863+0000 7f48e5fd9700 1 -- start start 2024-03-11T20:45:53.775 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.863+0000 7f48e5fd9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48e0105c50 0x7f48e0106070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:53.775 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.863+0000 7f48e5fd9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e0106640 con 0x7f48e0105c50 2024-03-11T20:45:53.775 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.863+0000 7f48df7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48e0105c50 0x7f48e0106070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:53.775 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.863+0000 7f48df7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48e0105c50 0x7f48e0106070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38982/0 (socket says 172.21.15.129:38982) 2024-03-11T20:45:53.775 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.863+0000 7f48df7fe700 1 -- 172.21.15.129:0/3504557500 learned_addr learned my addr 172.21.15.129:0/3504557500 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:53.775 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.864+0000 7f48df7fe700 1 -- 172.21.15.129:0/3504557500 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e0106e50 con 0x7f48e0105c50 2024-03-11T20:45:53.775 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.864+0000 7f48df7fe700 1 --2- 172.21.15.129:0/3504557500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48e0105c50 0x7f48e0106070 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f48d00098a0 tx=0x7f48d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=167059577a31f276 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:53.775 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.864+0000 7f48de7fc700 1 -- 172.21.15.129:0/3504557500 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48d0004160 con 0x7f48e0105c50 2024-03-11T20:45:53.775 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.864+0000 7f48de7fc700 1 -- 172.21.15.129:0/3504557500 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f48d0004300 con 0x7f48e0105c50 2024-03-11T20:45:53.775 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.864+0000 7f48de7fc700 1 -- 172.21.15.129:0/3504557500 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48d0011940 con 0x7f48e0105c50 2024-03-11T20:45:53.775 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.865+0000 7f48e5fd9700 1 -- 172.21.15.129:0/3504557500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48e0105c50 msgr2=0x7f48e0106070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:53.775 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.865+0000 7f48e5fd9700 1 --2- 172.21.15.129:0/3504557500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48e0105c50 0x7f48e0106070 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f48d00098a0 tx=0x7f48d0009bb0 comp rx=0 tx=0).stop 2024-03-11T20:45:53.775 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.865+0000 7f48e5fd9700 1 -- 172.21.15.129:0/3504557500 shutdown_connections 2024-03-11T20:45:53.776 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.865+0000 7f48e5fd9700 1 --2- 172.21.15.129:0/3504557500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48e0105c50 0x7f48e0106070 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:53.776 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.865+0000 7f48e5fd9700 1 -- 172.21.15.129:0/3504557500 >> 172.21.15.129:0/3504557500 conn(0x7f48e01011b0 msgr2=0x7f48e0103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:53.776 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.865+0000 7f48e5fd9700 1 -- 172.21.15.129:0/3504557500 shutdown_connections 2024-03-11T20:45:53.776 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.865+0000 7f48e5fd9700 1 -- 172.21.15.129:0/3504557500 wait complete. 2024-03-11T20:45:53.776 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.865+0000 7f48e5fd9700 1 Processor -- start 2024-03-11T20:45:53.776 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.865+0000 7f48e5fd9700 1 -- start start 2024-03-11T20:45:53.776 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.866+0000 7f48e5fd9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48e0105c50 0x7f48e0199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:53.776 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.866+0000 7f48e5fd9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e0107ce0 con 0x7f48e0105c50 2024-03-11T20:45:53.776 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.866+0000 7f48df7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48e0105c50 0x7f48e0199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:53.776 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.866+0000 7f48df7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48e0105c50 0x7f48e0199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38984/0 (socket says 172.21.15.129:38984) 2024-03-11T20:45:53.776 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.866+0000 7f48df7fe700 1 -- 172.21.15.129:0/917640208 learned_addr learned my addr 172.21.15.129:0/917640208 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:53.776 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.866+0000 7f48df7fe700 1 -- 172.21.15.129:0/917640208 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e0199f40 con 0x7f48e0105c50 2024-03-11T20:45:53.776 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.866+0000 7f48df7fe700 1 --2- 172.21.15.129:0/917640208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48e0105c50 0x7f48e0199a00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f48d0003e40 tx=0x7f48d00119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:53.776 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.866+0000 7f48dcff9700 1 -- 172.21.15.129:0/917640208 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48d001a430 con 0x7f48e0105c50 2024-03-11T20:45:53.776 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.866+0000 7f48e5fd9700 1 -- 172.21.15.129:0/917640208 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48e019a1d0 con 0x7f48e0105c50 2024-03-11T20:45:53.777 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.866+0000 7f48dcff9700 1 -- 172.21.15.129:0/917640208 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f48d001aa30 con 0x7f48e0105c50 2024-03-11T20:45:53.777 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.866+0000 7f48dcff9700 1 -- 172.21.15.129:0/917640208 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48d0024960 con 0x7f48e0105c50 2024-03-11T20:45:53.777 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.866+0000 7f48e5fd9700 1 -- 172.21.15.129:0/917640208 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e019cec0 con 0x7f48e0105c50 2024-03-11T20:45:53.777 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.867+0000 7f48dcff9700 1 -- 172.21.15.129:0/917640208 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f48d0022020 con 0x7f48e0105c50 2024-03-11T20:45:53.777 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.867+0000 7f48dcff9700 1 --2- 172.21.15.129:0/917640208 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f48c8038e80 0x7f48c803b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:53.777 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.867+0000 7f48e5fd9700 1 -- 172.21.15.129:0/917640208 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e0063410 con 0x7f48e0105c50 2024-03-11T20:45:53.777 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.867+0000 7f48dcff9700 1 -- 172.21.15.129:0/917640208 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f48d004d220 con 0x7f48e0105c50 2024-03-11T20:45:53.777 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.867+0000 7f48deffd700 1 --2- 172.21.15.129:0/917640208 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f48c8038e80 0x7f48c803b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:53.777 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.868+0000 7f48deffd700 1 --2- 172.21.15.129:0/917640208 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f48c8038e80 0x7f48c803b340 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f48d4006fd0 tx=0x7f48d4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:53.777 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:52.873+0000 7f48dcff9700 1 -- 172.21.15.129:0/917640208 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f48d00172b0 con 0x7f48e0105c50 2024-03-11T20:45:53.777 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:53.142+0000 7f48e5fd9700 1 -- 172.21.15.129:0/917640208 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f48e019d320 con 0x7f48e0105c50 2024-03-11T20:45:53.777 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:53.142+0000 7f48dcff9700 1 -- 172.21.15.129:0/917640208 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f48d0017940 con 0x7f48e0105c50 2024-03-11T20:45:53.777 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:53.144+0000 7f48e5fd9700 1 -- 172.21.15.129:0/917640208 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f48c8038e80 msgr2=0x7f48c803b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:53.777 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:53.144+0000 7f48e5fd9700 1 --2- 172.21.15.129:0/917640208 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f48c8038e80 0x7f48c803b340 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f48d4006fd0 tx=0x7f48d4006e40 comp rx=0 tx=0).stop 2024-03-11T20:45:53.778 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:53.144+0000 7f48e5fd9700 1 -- 172.21.15.129:0/917640208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48e0105c50 msgr2=0x7f48e0199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:53.778 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:53.144+0000 7f48e5fd9700 1 --2- 172.21.15.129:0/917640208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48e0105c50 0x7f48e0199a00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f48d0003e40 tx=0x7f48d00119c0 comp rx=0 tx=0).stop 2024-03-11T20:45:53.778 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:53.144+0000 7f48e5fd9700 1 -- 172.21.15.129:0/917640208 shutdown_connections 2024-03-11T20:45:53.778 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:53.144+0000 7f48e5fd9700 1 --2- 172.21.15.129:0/917640208 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f48c8038e80 0x7f48c803b340 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:53.778 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:53.144+0000 7f48e5fd9700 1 --2- 172.21.15.129:0/917640208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48e0105c50 0x7f48e0199a00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:53.778 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:53.144+0000 7f48e5fd9700 1 -- 172.21.15.129:0/917640208 >> 172.21.15.129:0/917640208 conn(0x7f48e01011b0 msgr2=0x7f48e0101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:53.778 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:53.145+0000 7f48e5fd9700 1 -- 172.21.15.129:0/917640208 shutdown_connections 2024-03-11T20:45:53.778 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:53.145+0000 7f48e5fd9700 1 -- 172.21.15.129:0/917640208 wait complete. 2024-03-11T20:45:53.778 INFO:teuthology.orchestra.run.smithi129.stdout:mgr is available 2024-03-11T20:45:54.687 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/320197412' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T20:45:55.080 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-03-11T20:45:55.080 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T20:45:55.080 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout fsid = 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:45:55.080 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T20:45:55.080 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T20:45:55.080 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T20:45:55.080 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T20:45:55.080 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-03-11T20:45:55.080 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T20:45:55.080 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T20:45:55.080 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 2024-03-11T20:45:55.081 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T20:45:55.081 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T20:45:55.081 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T20:45:55.081 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.204+0000 7f034c1ad700 1 Processor -- start 2024-03-11T20:45:55.081 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.204+0000 7f034c1ad700 1 -- start start 2024-03-11T20:45:55.081 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.204+0000 7f034c1ad700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034407b720 0x7f0344079b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:55.081 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.204+0000 7f034c1ad700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034407a150 con 0x7f034407b720 2024-03-11T20:45:55.081 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.205+0000 7f0349f49700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034407b720 0x7f0344079b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:55.081 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.205+0000 7f0349f49700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034407b720 0x7f0344079b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38990/0 (socket says 172.21.15.129:38990) 2024-03-11T20:45:55.081 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.205+0000 7f0349f49700 1 -- 172.21.15.129:0/3600070160 learned_addr learned my addr 172.21.15.129:0/3600070160 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:55.081 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.205+0000 7f0349f49700 1 -- 172.21.15.129:0/3600070160 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034407a2d0 con 0x7f034407b720 2024-03-11T20:45:55.081 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.205+0000 7f0349f49700 1 --2- 172.21.15.129:0/3600070160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034407b720 0x7f0344079b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f03400098a0 tx=0x7f0340009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16687ed9731e5002 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:55.081 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.205+0000 7f0348f47700 1 -- 172.21.15.129:0/3600070160 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0340004160 con 0x7f034407b720 2024-03-11T20:45:55.081 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.205+0000 7f0348f47700 1 -- 172.21.15.129:0/3600070160 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f0340004300 con 0x7f034407b720 2024-03-11T20:45:55.082 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.205+0000 7f0348f47700 1 -- 172.21.15.129:0/3600070160 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0340011940 con 0x7f034407b720 2024-03-11T20:45:55.082 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.206+0000 7f034c1ad700 1 -- 172.21.15.129:0/3600070160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034407b720 msgr2=0x7f0344079b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:55.082 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.206+0000 7f034c1ad700 1 --2- 172.21.15.129:0/3600070160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034407b720 0x7f0344079b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f03400098a0 tx=0x7f0340009bb0 comp rx=0 tx=0).stop 2024-03-11T20:45:55.082 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.206+0000 7f034c1ad700 1 -- 172.21.15.129:0/3600070160 shutdown_connections 2024-03-11T20:45:55.082 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.206+0000 7f034c1ad700 1 --2- 172.21.15.129:0/3600070160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034407b720 0x7f0344079b80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:55.082 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.206+0000 7f034c1ad700 1 -- 172.21.15.129:0/3600070160 >> 172.21.15.129:0/3600070160 conn(0x7f03441011f0 msgr2=0x7f0344103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:55.082 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.206+0000 7f034c1ad700 1 -- 172.21.15.129:0/3600070160 shutdown_connections 2024-03-11T20:45:55.082 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.206+0000 7f034c1ad700 1 -- 172.21.15.129:0/3600070160 wait complete. 2024-03-11T20:45:55.082 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.206+0000 7f034c1ad700 1 Processor -- start 2024-03-11T20:45:55.082 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.206+0000 7f034c1ad700 1 -- start start 2024-03-11T20:45:55.082 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.207+0000 7f034c1ad700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034407b720 0x7f034419bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:55.082 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.207+0000 7f034c1ad700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034410c090 con 0x7f034407b720 2024-03-11T20:45:55.083 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.207+0000 7f0349f49700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034407b720 0x7f034419bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:55.083 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.207+0000 7f0349f49700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034407b720 0x7f034419bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39004/0 (socket says 172.21.15.129:39004) 2024-03-11T20:45:55.083 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.207+0000 7f0349f49700 1 -- 172.21.15.129:0/320197412 learned_addr learned my addr 172.21.15.129:0/320197412 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:55.083 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.207+0000 7f0349f49700 1 -- 172.21.15.129:0/320197412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034419c160 con 0x7f034407b720 2024-03-11T20:45:55.083 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.207+0000 7f0349f49700 1 --2- 172.21.15.129:0/320197412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034407b720 0x7f034419bc20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0340009fb0 tx=0x7f03400119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:55.083 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.207+0000 7f033affd700 1 -- 172.21.15.129:0/320197412 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f034001a430 con 0x7f034407b720 2024-03-11T20:45:55.083 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.207+0000 7f033affd700 1 -- 172.21.15.129:0/320197412 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f034001aa30 con 0x7f034407b720 2024-03-11T20:45:55.083 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.207+0000 7f034c1ad700 1 -- 172.21.15.129:0/320197412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034419c3f0 con 0x7f034407b720 2024-03-11T20:45:55.083 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.207+0000 7f033affd700 1 -- 172.21.15.129:0/320197412 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0340024860 con 0x7f034407b720 2024-03-11T20:45:55.083 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.207+0000 7f034c1ad700 1 -- 172.21.15.129:0/320197412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034419f0e0 con 0x7f034407b720 2024-03-11T20:45:55.083 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.208+0000 7f033affd700 1 -- 172.21.15.129:0/320197412 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f0340022020 con 0x7f034407b720 2024-03-11T20:45:55.083 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.208+0000 7f033affd700 1 --2- 172.21.15.129:0/320197412 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f0330038ed0 0x7f033003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:55.083 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.208+0000 7f033affd700 1 -- 172.21.15.129:0/320197412 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f034004cf90 con 0x7f034407b720 2024-03-11T20:45:55.084 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.208+0000 7f0349748700 1 --2- 172.21.15.129:0/320197412 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f0330038ed0 0x7f033003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:55.084 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.208+0000 7f034c1ad700 1 -- 172.21.15.129:0/320197412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0344063410 con 0x7f034407b720 2024-03-11T20:45:55.084 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.209+0000 7f0349748700 1 --2- 172.21.15.129:0/320197412 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f0330038ed0 0x7f033003b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0334006fd0 tx=0x7f0334006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:55.084 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.213+0000 7f033affd700 1 -- 172.21.15.129:0/320197412 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f034000b3e0 con 0x7f034407b720 2024-03-11T20:45:55.084 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.405+0000 7f034c1ad700 1 -- 172.21.15.129:0/320197412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f03441a05c0 con 0x7f034407b720 2024-03-11T20:45:55.084 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.406+0000 7f033affd700 1 -- 172.21.15.129:0/320197412 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+282 (secure 0 0 0) 0x7f03400172b0 con 0x7f034407b720 2024-03-11T20:45:55.084 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.408+0000 7f034c1ad700 1 -- 172.21.15.129:0/320197412 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f0330038ed0 msgr2=0x7f033003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:55.084 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.408+0000 7f034c1ad700 1 --2- 172.21.15.129:0/320197412 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f0330038ed0 0x7f033003b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0334006fd0 tx=0x7f0334006e40 comp rx=0 tx=0).stop 2024-03-11T20:45:55.084 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.408+0000 7f034c1ad700 1 -- 172.21.15.129:0/320197412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034407b720 msgr2=0x7f034419bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:55.084 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.408+0000 7f034c1ad700 1 --2- 172.21.15.129:0/320197412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034407b720 0x7f034419bc20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0340009fb0 tx=0x7f03400119c0 comp rx=0 tx=0).stop 2024-03-11T20:45:55.084 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.408+0000 7f034c1ad700 1 -- 172.21.15.129:0/320197412 shutdown_connections 2024-03-11T20:45:55.084 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.408+0000 7f034c1ad700 1 --2- 172.21.15.129:0/320197412 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f0330038ed0 0x7f033003b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:55.084 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.408+0000 7f034c1ad700 1 --2- 172.21.15.129:0/320197412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f034407b720 0x7f034419bc20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:55.085 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.408+0000 7f034c1ad700 1 -- 172.21.15.129:0/320197412 >> 172.21.15.129:0/320197412 conn(0x7f03441011f0 msgr2=0x7f0344101d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:55.085 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.408+0000 7f034c1ad700 1 -- 172.21.15.129:0/320197412 shutdown_connections 2024-03-11T20:45:55.085 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:54.408+0000 7f034c1ad700 1 -- 172.21.15.129:0/320197412 wait complete. 2024-03-11T20:45:55.085 INFO:teuthology.orchestra.run.smithi129.stdout:Enabling cephadm module... 2024-03-11T20:45:56.169 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1031986257' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T20:45:56.169 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:55 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ignoring --setuser ceph since I am not root 2024-03-11T20:45:56.170 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:55 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ignoring --setgroup ceph since I am not root 2024-03-11T20:45:56.452 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.563+0000 7f5cb5cd6700 1 Processor -- start 2024-03-11T20:45:56.453 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.563+0000 7f5cb5cd6700 1 -- start start 2024-03-11T20:45:56.453 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.563+0000 7f5cb5cd6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cb0105c70 0x7f5cb0106090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:56.453 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.563+0000 7f5cb5cd6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb0106660 con 0x7f5cb0105c70 2024-03-11T20:45:56.453 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.563+0000 7f5caf7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cb0105c70 0x7f5cb0106090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:56.453 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.563+0000 7f5caf7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cb0105c70 0x7f5cb0106090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57970/0 (socket says 172.21.15.129:57970) 2024-03-11T20:45:56.453 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.563+0000 7f5caf7fe700 1 -- 172.21.15.129:0/3223399241 learned_addr learned my addr 172.21.15.129:0/3223399241 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:56.453 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.564+0000 7f5caf7fe700 1 -- 172.21.15.129:0/3223399241 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb0106e70 con 0x7f5cb0105c70 2024-03-11T20:45:56.453 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.564+0000 7f5caf7fe700 1 --2- 172.21.15.129:0/3223399241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cb0105c70 0x7f5cb0106090 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5ca00098a0 tx=0x7f5ca0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172cf555b48a4649 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:56.453 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.564+0000 7f5cae7fc700 1 -- 172.21.15.129:0/3223399241 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ca0004160 con 0x7f5cb0105c70 2024-03-11T20:45:56.453 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.564+0000 7f5cae7fc700 1 -- 172.21.15.129:0/3223399241 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f5ca0004300 con 0x7f5cb0105c70 2024-03-11T20:45:56.453 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.564+0000 7f5cae7fc700 1 -- 172.21.15.129:0/3223399241 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ca0011940 con 0x7f5cb0105c70 2024-03-11T20:45:56.453 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.565+0000 7f5cb5cd6700 1 -- 172.21.15.129:0/3223399241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cb0105c70 msgr2=0x7f5cb0106090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:56.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.565+0000 7f5cb5cd6700 1 --2- 172.21.15.129:0/3223399241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cb0105c70 0x7f5cb0106090 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5ca00098a0 tx=0x7f5ca0009bb0 comp rx=0 tx=0).stop 2024-03-11T20:45:56.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.565+0000 7f5cb5cd6700 1 -- 172.21.15.129:0/3223399241 shutdown_connections 2024-03-11T20:45:56.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.565+0000 7f5cb5cd6700 1 --2- 172.21.15.129:0/3223399241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cb0105c70 0x7f5cb0106090 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:56.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.565+0000 7f5cb5cd6700 1 -- 172.21.15.129:0/3223399241 >> 172.21.15.129:0/3223399241 conn(0x7f5cb01011f0 msgr2=0x7f5cb0103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:56.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.565+0000 7f5cb5cd6700 1 -- 172.21.15.129:0/3223399241 shutdown_connections 2024-03-11T20:45:56.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.565+0000 7f5cb5cd6700 1 -- 172.21.15.129:0/3223399241 wait complete. 2024-03-11T20:45:56.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.565+0000 7f5cb5cd6700 1 Processor -- start 2024-03-11T20:45:56.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.565+0000 7f5cb5cd6700 1 -- start start 2024-03-11T20:45:56.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.566+0000 7f5cb5cd6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cb0105c70 0x7f5cb0199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:56.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.566+0000 7f5cb5cd6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb0107d00 con 0x7f5cb0105c70 2024-03-11T20:45:56.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.566+0000 7f5caf7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cb0105c70 0x7f5cb0199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:56.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.566+0000 7f5caf7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cb0105c70 0x7f5cb0199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57978/0 (socket says 172.21.15.129:57978) 2024-03-11T20:45:56.454 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.566+0000 7f5caf7fe700 1 -- 172.21.15.129:0/1031986257 learned_addr learned my addr 172.21.15.129:0/1031986257 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:56.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.566+0000 7f5caf7fe700 1 -- 172.21.15.129:0/1031986257 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb0199f40 con 0x7f5cb0105c70 2024-03-11T20:45:56.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.566+0000 7f5caf7fe700 1 --2- 172.21.15.129:0/1031986257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cb0105c70 0x7f5cb0199a00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0003e40 tx=0x7f5ca00119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:56.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.566+0000 7f5cacff9700 1 -- 172.21.15.129:0/1031986257 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ca001a430 con 0x7f5cb0105c70 2024-03-11T20:45:56.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.566+0000 7f5cacff9700 1 -- 172.21.15.129:0/1031986257 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f5ca001aa30 con 0x7f5cb0105c70 2024-03-11T20:45:56.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.566+0000 7f5cb5cd6700 1 -- 172.21.15.129:0/1031986257 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb019a1d0 con 0x7f5cb0105c70 2024-03-11T20:45:56.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.566+0000 7f5cacff9700 1 -- 172.21.15.129:0/1031986257 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ca0024960 con 0x7f5cb0105c70 2024-03-11T20:45:56.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.566+0000 7f5cb5cd6700 1 -- 172.21.15.129:0/1031986257 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb019cec0 con 0x7f5cb0105c70 2024-03-11T20:45:56.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.567+0000 7f5cb5cd6700 1 -- 172.21.15.129:0/1031986257 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb0063410 con 0x7f5cb0105c70 2024-03-11T20:45:56.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.567+0000 7f5cacff9700 1 -- 172.21.15.129:0/1031986257 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f5ca0022020 con 0x7f5cb0105c70 2024-03-11T20:45:56.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.568+0000 7f5cacff9700 1 --2- 172.21.15.129:0/1031986257 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f5c98038e80 0x7f5c9803b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:56.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.568+0000 7f5cacff9700 1 -- 172.21.15.129:0/1031986257 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5ca004d3d0 con 0x7f5cb0105c70 2024-03-11T20:45:56.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.568+0000 7f5caeffd700 1 --2- 172.21.15.129:0/1031986257 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f5c98038e80 0x7f5c9803b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:56.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.568+0000 7f5caeffd700 1 --2- 172.21.15.129:0/1031986257 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f5c98038e80 0x7f5c9803b340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4006fd0 tx=0x7f5ca4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:56.455 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.572+0000 7f5cacff9700 1 -- 172.21.15.129:0/1031986257 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f5ca000b3e0 con 0x7f5cb0105c70 2024-03-11T20:45:56.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.822+0000 7f5cb5cd6700 1 -- 172.21.15.129:0/1031986257 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f5cb019d320 con 0x7f5cb0105c70 2024-03-11T20:45:56.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.874+0000 7f5cacff9700 1 -- 172.21.15.129:0/1031986257 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f5ca0004300 con 0x7f5cb0105c70 2024-03-11T20:45:56.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.875+0000 7f5cacff9700 1 -- 172.21.15.129:0/1031986257 <== mon.0 v2:172.21.15.129:3300/0 8 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f5ca002dc90 con 0x7f5cb0105c70 2024-03-11T20:45:56.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.876+0000 7f5cb5cd6700 1 -- 172.21.15.129:0/1031986257 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f5c98038e80 msgr2=0x7f5c9803b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:56.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.876+0000 7f5cb5cd6700 1 --2- 172.21.15.129:0/1031986257 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f5c98038e80 0x7f5c9803b340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4006fd0 tx=0x7f5ca4006e40 comp rx=0 tx=0).stop 2024-03-11T20:45:56.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.876+0000 7f5cb5cd6700 1 -- 172.21.15.129:0/1031986257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cb0105c70 msgr2=0x7f5cb0199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:56.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.876+0000 7f5cb5cd6700 1 --2- 172.21.15.129:0/1031986257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cb0105c70 0x7f5cb0199a00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0003e40 tx=0x7f5ca00119c0 comp rx=0 tx=0).stop 2024-03-11T20:45:56.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.876+0000 7f5cb5cd6700 1 -- 172.21.15.129:0/1031986257 shutdown_connections 2024-03-11T20:45:56.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.876+0000 7f5cb5cd6700 1 --2- 172.21.15.129:0/1031986257 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f5c98038e80 0x7f5c9803b340 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:56.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.876+0000 7f5cb5cd6700 1 --2- 172.21.15.129:0/1031986257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cb0105c70 0x7f5cb0199a00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:56.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.876+0000 7f5cb5cd6700 1 -- 172.21.15.129:0/1031986257 >> 172.21.15.129:0/1031986257 conn(0x7f5cb01011f0 msgr2=0x7f5cb0101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:56.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.877+0000 7f5cb5cd6700 1 -- 172.21.15.129:0/1031986257 shutdown_connections 2024-03-11T20:45:56.456 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:55.877+0000 7f5cb5cd6700 1 -- 172.21.15.129:0/1031986257 wait complete. 2024-03-11T20:45:56.698 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:56.426+0000 7fca34db0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T20:45:56.699 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:56.524+0000 7fca34db0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T20:45:56.951 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1031986257' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T20:45:56.951 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:56 smithi129 ceph-mon[30184]: mgrmap e5: y(active, since 6s) 2024-03-11T20:45:57.222 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:57 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:57.179+0000 7fca34db0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T20:45:57.476 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:57 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:57.284+0000 7fca34db0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T20:45:57.477 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:57 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:57.426+0000 7fca34db0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T20:45:57.834 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:57 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:57.522+0000 7fca34db0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T20:45:57.834 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:57 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:57.603+0000 7fca34db0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T20:45:57.842 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout { 2024-03-11T20:45:57.843 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-11T20:45:57.843 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T20:45:57.843 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T20:45:57.843 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T20:45:57.843 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout } 2024-03-11T20:45:57.843 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.955+0000 7fd04c54f700 1 Processor -- start 2024-03-11T20:45:57.843 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.955+0000 7fd04c54f700 1 -- start start 2024-03-11T20:45:57.843 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.956+0000 7fd04c54f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd044106140 0x7fd044106560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:57.843 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.956+0000 7fd04c54f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd044106b30 con 0x7fd044106140 2024-03-11T20:45:57.843 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.956+0000 7fd04a2eb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd044106140 0x7fd044106560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:57.843 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.956+0000 7fd04a2eb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd044106140 0x7fd044106560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57990/0 (socket says 172.21.15.129:57990) 2024-03-11T20:45:57.843 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.956+0000 7fd04a2eb700 1 -- 172.21.15.129:0/2459367272 learned_addr learned my addr 172.21.15.129:0/2459367272 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:57.843 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.956+0000 7fd04a2eb700 1 -- 172.21.15.129:0/2459367272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd044106cb0 con 0x7fd044106140 2024-03-11T20:45:57.843 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.956+0000 7fd04a2eb700 1 --2- 172.21.15.129:0/2459367272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd044106140 0x7fd044106560 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd0400098a0 tx=0x7fd040009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c04bd73c651e065 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:57.844 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.956+0000 7fd0492e9700 1 -- 172.21.15.129:0/2459367272 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd040004160 con 0x7fd044106140 2024-03-11T20:45:57.844 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.957+0000 7fd0492e9700 1 -- 172.21.15.129:0/2459367272 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fd040004300 con 0x7fd044106140 2024-03-11T20:45:57.844 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.957+0000 7fd0492e9700 1 -- 172.21.15.129:0/2459367272 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd040011940 con 0x7fd044106140 2024-03-11T20:45:57.844 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.957+0000 7fd04c54f700 1 -- 172.21.15.129:0/2459367272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd044106140 msgr2=0x7fd044106560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:57.844 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.957+0000 7fd04c54f700 1 --2- 172.21.15.129:0/2459367272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd044106140 0x7fd044106560 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd0400098a0 tx=0x7fd040009bb0 comp rx=0 tx=0).stop 2024-03-11T20:45:57.844 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.957+0000 7fd04c54f700 1 -- 172.21.15.129:0/2459367272 shutdown_connections 2024-03-11T20:45:57.844 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.957+0000 7fd04c54f700 1 --2- 172.21.15.129:0/2459367272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd044106140 0x7fd044106560 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:57.844 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.957+0000 7fd04c54f700 1 -- 172.21.15.129:0/2459367272 >> 172.21.15.129:0/2459367272 conn(0x7fd0440ff850 msgr2=0x7fd044101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:57.844 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.958+0000 7fd04c54f700 1 -- 172.21.15.129:0/2459367272 shutdown_connections 2024-03-11T20:45:57.844 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.958+0000 7fd04c54f700 1 -- 172.21.15.129:0/2459367272 wait complete. 2024-03-11T20:45:57.844 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.958+0000 7fd04c54f700 1 Processor -- start 2024-03-11T20:45:57.844 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.958+0000 7fd04c54f700 1 -- start start 2024-03-11T20:45:57.845 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.958+0000 7fd04c54f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd044106140 0x7fd04419bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:57.845 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.958+0000 7fd04c54f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd044104920 con 0x7fd044106140 2024-03-11T20:45:57.845 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.958+0000 7fd04a2eb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd044106140 0x7fd04419bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:45:57.845 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.958+0000 7fd04a2eb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd044106140 0x7fd04419bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58006/0 (socket says 172.21.15.129:58006) 2024-03-11T20:45:57.845 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.958+0000 7fd04a2eb700 1 -- 172.21.15.129:0/1430745234 learned_addr learned my addr 172.21.15.129:0/1430745234 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:45:57.845 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.959+0000 7fd04a2eb700 1 -- 172.21.15.129:0/1430745234 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04419c1a0 con 0x7fd044106140 2024-03-11T20:45:57.845 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.959+0000 7fd04a2eb700 1 --2- 172.21.15.129:0/1430745234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd044106140 0x7fd04419bc60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd040003f40 tx=0x7fd040011910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:45:57.845 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.959+0000 7fd03b7fe700 1 -- 172.21.15.129:0/1430745234 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0400039c0 con 0x7fd044106140 2024-03-11T20:45:57.845 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.959+0000 7fd03b7fe700 1 -- 172.21.15.129:0/1430745234 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fd040024890 con 0x7fd044106140 2024-03-11T20:45:57.845 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.959+0000 7fd04c54f700 1 -- 172.21.15.129:0/1430745234 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd04419c490 con 0x7fd044106140 2024-03-11T20:45:57.845 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.959+0000 7fd03b7fe700 1 -- 172.21.15.129:0/1430745234 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd04001a6a0 con 0x7fd044106140 2024-03-11T20:45:57.845 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.959+0000 7fd04c54f700 1 -- 172.21.15.129:0/1430745234 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04419f0d0 con 0x7fd044106140 2024-03-11T20:45:57.845 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.960+0000 7fd03b7fe700 1 -- 172.21.15.129:0/1430745234 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fd04001a970 con 0x7fd044106140 2024-03-11T20:45:57.846 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.960+0000 7fd04c54f700 1 -- 172.21.15.129:0/1430745234 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd044063410 con 0x7fd044106140 2024-03-11T20:45:57.846 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.960+0000 7fd03b7fe700 1 --2- 172.21.15.129:0/1430745234 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7fd030038f20 0x7fd03003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:45:57.846 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.960+0000 7fd03b7fe700 1 -- 172.21.15.129:0/1430745234 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd040024a40 con 0x7fd044106140 2024-03-11T20:45:57.846 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.960+0000 7fd049aea700 1 -- 172.21.15.129:0/1430745234 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7fd030038f20 msgr2=0x7fd03003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2213467472 2024-03-11T20:45:57.846 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.960+0000 7fd049aea700 1 --2- 172.21.15.129:0/1430745234 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7fd030038f20 0x7fd03003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T20:45:57.846 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:56.966+0000 7fd03b7fe700 1 -- 172.21.15.129:0/1430745234 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fd04002c9f0 con 0x7fd044106140 2024-03-11T20:45:57.846 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:57.161+0000 7fd049aea700 1 -- 172.21.15.129:0/1430745234 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7fd030038f20 msgr2=0x7fd03003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2213467472 2024-03-11T20:45:57.846 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:57.161+0000 7fd049aea700 1 --2- 172.21.15.129:0/1430745234 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7fd030038f20 0x7fd03003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T20:45:57.846 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:57.227+0000 7fd04c54f700 1 -- 172.21.15.129:0/1430745234 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd0441951c0 con 0x7fd044106140 2024-03-11T20:45:57.846 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:57.227+0000 7fd03b7fe700 1 -- 172.21.15.129:0/1430745234 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fd0441951c0 con 0x7fd044106140 2024-03-11T20:45:57.846 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:57.229+0000 7fd04c54f700 1 -- 172.21.15.129:0/1430745234 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7fd030038f20 msgr2=0x7fd03003b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T20:45:57.847 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:57.229+0000 7fd04c54f700 1 --2- 172.21.15.129:0/1430745234 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7fd030038f20 0x7fd03003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:57.847 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:57.229+0000 7fd04c54f700 1 -- 172.21.15.129:0/1430745234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd044106140 msgr2=0x7fd04419bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:45:57.847 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:57.229+0000 7fd04c54f700 1 --2- 172.21.15.129:0/1430745234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd044106140 0x7fd04419bc60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd040003f40 tx=0x7fd040011910 comp rx=0 tx=0).stop 2024-03-11T20:45:57.847 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:57.230+0000 7fd04c54f700 1 -- 172.21.15.129:0/1430745234 shutdown_connections 2024-03-11T20:45:57.847 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:57.230+0000 7fd04c54f700 1 --2- 172.21.15.129:0/1430745234 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7fd030038f20 0x7fd03003b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:57.847 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:57.230+0000 7fd04c54f700 1 --2- 172.21.15.129:0/1430745234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd044106140 0x7fd04419bc60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:45:57.847 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:57.230+0000 7fd04c54f700 1 -- 172.21.15.129:0/1430745234 >> 172.21.15.129:0/1430745234 conn(0x7fd0440ff850 msgr2=0x7fd044192280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:45:57.847 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:57.230+0000 7fd04c54f700 1 -- 172.21.15.129:0/1430745234 shutdown_connections 2024-03-11T20:45:57.847 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:57.230+0000 7fd04c54f700 1 -- 172.21.15.129:0/1430745234 wait complete. 2024-03-11T20:45:57.847 INFO:teuthology.orchestra.run.smithi129.stdout:Waiting for the mgr to restart... 2024-03-11T20:45:57.847 INFO:teuthology.orchestra.run.smithi129.stdout:Waiting for mgr epoch 5... 2024-03-11T20:45:58.098 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:45:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1430745234' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T20:45:58.098 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:57 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:57.872+0000 7fca34db0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T20:45:58.098 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:57 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:57.984+0000 7fca34db0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T20:45:58.098 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:58 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:58.070+0000 7fca34db0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T20:45:58.357 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:58 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:58.170+0000 7fca34db0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T20:45:58.357 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:58 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:58.348+0000 7fca34db0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T20:45:59.727 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:59 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:59.305+0000 7fca34db0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T20:45:59.727 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:59 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:59.434+0000 7fca34db0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T20:46:00.054 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:59 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:59.773+0000 7fca34db0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T20:46:00.054 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:45:59 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:45:59.864+0000 7fca34db0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T20:46:00.054 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:00 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:00.058+0000 7fca34db0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:46:00.477 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:00 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:00.237+0000 7fca34db0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T20:46:00.766 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:00 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:00.514+0000 7fca34db0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:46:00.767 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:00 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:00.771+0000 7fca34db0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T20:46:01.227 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:01.001+0000 7fca34db0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T20:46:02.227 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:01.786+0000 7fca34db0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T20:46:02.652 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:02.348+0000 7fca34db0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T20:46:02.652 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:02.444+0000 7fca34db0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T20:46:02.652 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:02.656+0000 7fca34db0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T20:46:02.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-mon[30184]: Active manager daemon y restarted 2024-03-11T20:46:02.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-mon[30184]: Activating manager daemon y 2024-03-11T20:46:02.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-mon[30184]: osdmap e2: 0 total, 0 up, 0 in 2024-03-11T20:46:02.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-mon[30184]: mgrmap e6: y(active, starting, since 0.00337125s) 2024-03-11T20:46:02.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:46:02.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T20:46:02.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T20:46:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T20:46:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T20:46:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-mon[30184]: Manager daemon y is now available 2024-03-11T20:46:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' 2024-03-11T20:46:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' 2024-03-11T20:46:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:02 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:03.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:03 smithi129 ceph-mon[30184]: Found migration_current of "None". Setting to last migration. 2024-03-11T20:46:03.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:03 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:03.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:03 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T20:46:03.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:03 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T20:46:03.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:03 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' 2024-03-11T20:46:03.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:03 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' 2024-03-11T20:46:03.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:03 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:03.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:03 smithi129 ceph-mon[30184]: mgrmap e7: y(active, since 1.00575s) 2024-03-11T20:46:04.209 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout { 2024-03-11T20:46:04.209 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-11T20:46:04.209 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T20:46:04.209 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout } 2024-03-11T20:46:04.210 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.361+0000 7f0514414700 1 Processor -- start 2024-03-11T20:46:04.210 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.361+0000 7f0514414700 1 -- start start 2024-03-11T20:46:04.210 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.361+0000 7f0514414700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c105c70 0x7f050c106090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:04.210 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.361+0000 7f0514414700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c106660 con 0x7f050c105c70 2024-03-11T20:46:04.210 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.361+0000 7f05121b0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c105c70 0x7f050c106090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:04.210 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.362+0000 7f05121b0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c105c70 0x7f050c106090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58018/0 (socket says 172.21.15.129:58018) 2024-03-11T20:46:04.210 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.362+0000 7f05121b0700 1 -- 172.21.15.129:0/2589192742 learned_addr learned my addr 172.21.15.129:0/2589192742 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:04.210 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.362+0000 7f05121b0700 1 -- 172.21.15.129:0/2589192742 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c106e70 con 0x7f050c105c70 2024-03-11T20:46:04.210 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.362+0000 7f05121b0700 1 --2- 172.21.15.129:0/2589192742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c105c70 0x7f050c106090 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f04fc0098a0 tx=0x7f04fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2b87b9490989bf server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:04.210 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.362+0000 7f05111ae700 1 -- 172.21.15.129:0/2589192742 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04fc004160 con 0x7f050c105c70 2024-03-11T20:46:04.210 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.362+0000 7f05111ae700 1 -- 172.21.15.129:0/2589192742 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f04fc004300 con 0x7f050c105c70 2024-03-11T20:46:04.210 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.362+0000 7f05111ae700 1 -- 172.21.15.129:0/2589192742 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04fc011940 con 0x7f050c105c70 2024-03-11T20:46:04.210 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.363+0000 7f0514414700 1 -- 172.21.15.129:0/2589192742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c105c70 msgr2=0x7f050c106090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:04.211 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.363+0000 7f0514414700 1 --2- 172.21.15.129:0/2589192742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c105c70 0x7f050c106090 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f04fc0098a0 tx=0x7f04fc009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:04.211 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.363+0000 7f0514414700 1 -- 172.21.15.129:0/2589192742 shutdown_connections 2024-03-11T20:46:04.211 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.363+0000 7f0514414700 1 --2- 172.21.15.129:0/2589192742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c105c70 0x7f050c106090 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:04.211 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.363+0000 7f0514414700 1 -- 172.21.15.129:0/2589192742 >> 172.21.15.129:0/2589192742 conn(0x7f050c1011f0 msgr2=0x7f050c103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:04.211 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.363+0000 7f0514414700 1 -- 172.21.15.129:0/2589192742 shutdown_connections 2024-03-11T20:46:04.211 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.363+0000 7f0514414700 1 -- 172.21.15.129:0/2589192742 wait complete. 2024-03-11T20:46:04.211 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.364+0000 7f0514414700 1 Processor -- start 2024-03-11T20:46:04.211 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.364+0000 7f0514414700 1 -- start start 2024-03-11T20:46:04.211 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.364+0000 7f0514414700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c105c70 0x7f050c199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:04.211 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.364+0000 7f0514414700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c107d00 con 0x7f050c105c70 2024-03-11T20:46:04.211 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.364+0000 7f05121b0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c105c70 0x7f050c199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:04.211 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.364+0000 7f05121b0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c105c70 0x7f050c199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58030/0 (socket says 172.21.15.129:58030) 2024-03-11T20:46:04.211 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.364+0000 7f05121b0700 1 -- 172.21.15.129:0/3917521463 learned_addr learned my addr 172.21.15.129:0/3917521463 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:04.211 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.365+0000 7f05121b0700 1 -- 172.21.15.129:0/3917521463 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c199f80 con 0x7f050c105c70 2024-03-11T20:46:04.212 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.365+0000 7f05121b0700 1 --2- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c105c70 0x7f050c199a40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009fb0 tx=0x7f04fc0119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:04.212 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.365+0000 7f05037fe700 1 -- 172.21.15.129:0/3917521463 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04fc01a430 con 0x7f050c105c70 2024-03-11T20:46:04.212 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.365+0000 7f05037fe700 1 -- 172.21.15.129:0/3917521463 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f04fc01aa30 con 0x7f050c105c70 2024-03-11T20:46:04.212 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.365+0000 7f05037fe700 1 -- 172.21.15.129:0/3917521463 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f04fc024860 con 0x7f050c105c70 2024-03-11T20:46:04.212 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.365+0000 7f0514414700 1 -- 172.21.15.129:0/3917521463 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050c19a210 con 0x7f050c105c70 2024-03-11T20:46:04.212 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.365+0000 7f0514414700 1 -- 172.21.15.129:0/3917521463 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f050c19cf00 con 0x7f050c105c70 2024-03-11T20:46:04.212 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.366+0000 7f05037fe700 1 -- 172.21.15.129:0/3917521463 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f04fc022020 con 0x7f050c105c70 2024-03-11T20:46:04.212 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.366+0000 7f05037fe700 1 --2- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f04f8038f20 0x7f04f803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:04.212 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.366+0000 7f05037fe700 1 -- 172.21.15.129:0/3917521463 --> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04f803baf0 con 0x7f04f8038f20 2024-03-11T20:46:04.212 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.367+0000 7f05037fe700 1 -- 172.21.15.129:0/3917521463 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f04fc04cff0 con 0x7f050c105c70 2024-03-11T20:46:04.212 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.367+0000 7f05119af700 1 -- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f04f8038f20 msgr2=0x7f04f803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2213467472 2024-03-11T20:46:04.212 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.367+0000 7f05119af700 1 --2- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f04f8038f20 0x7f04f803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T20:46:04.212 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.568+0000 7f05119af700 1 -- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f04f8038f20 msgr2=0x7f04f803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2213467472 2024-03-11T20:46:04.213 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.568+0000 7f05119af700 1 --2- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f04f8038f20 0x7f04f803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T20:46:04.213 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.969+0000 7f05119af700 1 -- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f04f8038f20 msgr2=0x7f04f803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2213467472 2024-03-11T20:46:04.213 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:58.969+0000 7f05119af700 1 --2- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f04f8038f20 0x7f04f803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-11T20:46:04.213 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:59.770+0000 7f05119af700 1 -- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f04f8038f20 msgr2=0x7f04f803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2213467472 2024-03-11T20:46:04.213 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:45:59.770+0000 7f05119af700 1 --2- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f04f8038f20 0x7f04f803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-11T20:46:04.213 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:01.371+0000 7f05119af700 1 -- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f04f8038f20 msgr2=0x7f04f803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2213467472 2024-03-11T20:46:04.213 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:01.371+0000 7f05119af700 1 --2- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f04f8038f20 0x7f04f803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-11T20:46:04.213 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:02.672+0000 7f05037fe700 1 -- 172.21.15.129:0/3917521463 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7f04fc02d3f0 con 0x7f050c105c70 2024-03-11T20:46:04.213 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:02.672+0000 7f05037fe700 1 -- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f04f8038f20 msgr2=0x7f04f803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T20:46:04.213 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:02.672+0000 7f05037fe700 1 --2- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f04f8038f20 0x7f04f803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:04.213 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.674+0000 7f05037fe700 1 -- 172.21.15.129:0/3917521463 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7f04fc04ee00 con 0x7f050c105c70 2024-03-11T20:46:04.213 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.674+0000 7f05037fe700 1 --2- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f04f803c810 0x7f04f803ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:04.213 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.674+0000 7f05037fe700 1 -- 172.21.15.129:0/3917521463 --> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04f803baf0 con 0x7f04f803c810 2024-03-11T20:46:04.213 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.675+0000 7f05119af700 1 --2- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f04f803c810 0x7f04f803ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:04.214 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.676+0000 7f05119af700 1 --2- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f04f803c810 0x7f04f803ec00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0508003a10 tx=0x7f0508009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:04.214 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.676+0000 7f05037fe700 1 -- 172.21.15.129:0/3917521463 <== mgr.14120 v2:172.21.15.129:6800/521270824 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f04f803baf0 con 0x7f04f803c810 2024-03-11T20:46:04.214 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.684+0000 7f0514414700 1 -- 172.21.15.129:0/3917521463 --> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f050c063410 con 0x7f04f803c810 2024-03-11T20:46:04.214 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.684+0000 7f05037fe700 1 -- 172.21.15.129:0/3917521463 <== mgr.14120 v2:172.21.15.129:6800/521270824 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f050c063410 con 0x7f04f803c810 2024-03-11T20:46:04.214 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.684+0000 7f0514414700 1 -- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f04f803c810 msgr2=0x7f04f803ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:04.214 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.684+0000 7f0514414700 1 --2- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f04f803c810 0x7f04f803ec00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0508003a10 tx=0x7f0508009290 comp rx=0 tx=0).stop 2024-03-11T20:46:04.214 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.684+0000 7f0514414700 1 -- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c105c70 msgr2=0x7f050c199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:04.214 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.684+0000 7f0514414700 1 --2- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c105c70 0x7f050c199a40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009fb0 tx=0x7f04fc0119c0 comp rx=0 tx=0).stop 2024-03-11T20:46:04.214 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.684+0000 7f0514414700 1 -- 172.21.15.129:0/3917521463 shutdown_connections 2024-03-11T20:46:04.214 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.684+0000 7f0514414700 1 --2- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f04f803c810 0x7f04f803ec00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:04.214 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.684+0000 7f0514414700 1 --2- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f050c105c70 0x7f050c199a40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:04.214 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.684+0000 7f0514414700 1 --2- 172.21.15.129:0/3917521463 >> [v2:172.21.15.129:6800/2213467472,v1:172.21.15.129:6801/2213467472] conn(0x7f04f8038f20 0x7f04f803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:04.215 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.684+0000 7f0514414700 1 -- 172.21.15.129:0/3917521463 >> 172.21.15.129:0/3917521463 conn(0x7f050c1011f0 msgr2=0x7f050c101d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:04.215 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.684+0000 7f0514414700 1 -- 172.21.15.129:0/3917521463 shutdown_connections 2024-03-11T20:46:04.215 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:03.684+0000 7f0514414700 1 -- 172.21.15.129:0/3917521463 wait complete. 2024-03-11T20:46:04.215 INFO:teuthology.orchestra.run.smithi129.stdout:mgr epoch 5 is available 2024-03-11T20:46:04.215 INFO:teuthology.orchestra.run.smithi129.stdout:Setting orchestrator backend to cephadm... 2024-03-11T20:46:04.918 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:04 smithi129 ceph-mon[30184]: [11/Mar/2024:20:46:03] ENGINE Bus STARTING 2024-03-11T20:46:04.919 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:04 smithi129 ceph-mon[30184]: [11/Mar/2024:20:46:03] ENGINE Serving on https://172.21.15.129:7150 2024-03-11T20:46:04.919 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:04 smithi129 ceph-mon[30184]: [11/Mar/2024:20:46:03] ENGINE Serving on http://172.21.15.129:8765 2024-03-11T20:46:04.919 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:04 smithi129 ceph-mon[30184]: [11/Mar/2024:20:46:03] ENGINE Bus STARTED 2024-03-11T20:46:04.919 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:04 smithi129 ceph-mon[30184]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T20:46:04.919 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:04 smithi129 ceph-mon[30184]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T20:46:05.471 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.700+0000 7f809236d700 1 Processor -- start 2024-03-11T20:46:05.471 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.700+0000 7f809236d700 1 -- start start 2024-03-11T20:46:05.472 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.700+0000 7f809236d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f808c104b30 0x7f808c104f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:05.472 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.700+0000 7f809236d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808c105520 con 0x7f808c104b30 2024-03-11T20:46:05.472 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.701+0000 7f808bfff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f808c104b30 0x7f808c104f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:05.472 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.701+0000 7f808bfff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f808c104b30 0x7f808c104f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58088/0 (socket says 172.21.15.129:58088) 2024-03-11T20:46:05.472 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.701+0000 7f808bfff700 1 -- 172.21.15.129:0/3896795007 learned_addr learned my addr 172.21.15.129:0/3896795007 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:05.472 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.701+0000 7f808bfff700 1 -- 172.21.15.129:0/3896795007 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808c105d30 con 0x7f808c104b30 2024-03-11T20:46:05.472 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.701+0000 7f808bfff700 1 --2- 172.21.15.129:0/3896795007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f808c104b30 0x7f808c104f50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f807c0098a0 tx=0x7f807c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00842991c7a4cfb server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:05.472 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.701+0000 7f808affd700 1 -- 172.21.15.129:0/3896795007 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f807c004160 con 0x7f808c104b30 2024-03-11T20:46:05.472 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.701+0000 7f808affd700 1 -- 172.21.15.129:0/3896795007 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f807c004300 con 0x7f808c104b30 2024-03-11T20:46:05.472 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.701+0000 7f808affd700 1 -- 172.21.15.129:0/3896795007 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f807c011940 con 0x7f808c104b30 2024-03-11T20:46:05.472 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.702+0000 7f809236d700 1 -- 172.21.15.129:0/3896795007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f808c104b30 msgr2=0x7f808c104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:05.472 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.702+0000 7f809236d700 1 --2- 172.21.15.129:0/3896795007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f808c104b30 0x7f808c104f50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f807c0098a0 tx=0x7f807c009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:05.472 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.702+0000 7f809236d700 1 -- 172.21.15.129:0/3896795007 shutdown_connections 2024-03-11T20:46:05.472 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.702+0000 7f809236d700 1 --2- 172.21.15.129:0/3896795007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f808c104b30 0x7f808c104f50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:05.473 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.702+0000 7f809236d700 1 -- 172.21.15.129:0/3896795007 >> 172.21.15.129:0/3896795007 conn(0x7f808c1000b0 msgr2=0x7f808c102510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:05.473 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.702+0000 7f809236d700 1 -- 172.21.15.129:0/3896795007 shutdown_connections 2024-03-11T20:46:05.473 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.702+0000 7f809236d700 1 -- 172.21.15.129:0/3896795007 wait complete. 2024-03-11T20:46:05.473 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.702+0000 7f809236d700 1 Processor -- start 2024-03-11T20:46:05.473 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.702+0000 7f809236d700 1 -- start start 2024-03-11T20:46:05.473 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.703+0000 7f809236d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f808c104b30 0x7f808c1977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:05.473 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.703+0000 7f809236d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808c106b00 con 0x7f808c104b30 2024-03-11T20:46:05.473 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.703+0000 7f808bfff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f808c104b30 0x7f808c1977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:05.473 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.703+0000 7f808bfff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f808c104b30 0x7f808c1977f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58096/0 (socket says 172.21.15.129:58096) 2024-03-11T20:46:05.473 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.703+0000 7f808bfff700 1 -- 172.21.15.129:0/507189561 learned_addr learned my addr 172.21.15.129:0/507189561 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:05.473 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.703+0000 7f808bfff700 1 -- 172.21.15.129:0/507189561 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808c197d30 con 0x7f808c104b30 2024-03-11T20:46:05.473 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.703+0000 7f808bfff700 1 --2- 172.21.15.129:0/507189561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f808c104b30 0x7f808c1977f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f807c003e40 tx=0x7f807c0119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:05.473 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.703+0000 7f80897fa700 1 -- 172.21.15.129:0/507189561 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f807c01a430 con 0x7f808c104b30 2024-03-11T20:46:05.473 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.703+0000 7f809236d700 1 -- 172.21.15.129:0/507189561 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f808c197fc0 con 0x7f808c104b30 2024-03-11T20:46:05.474 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.703+0000 7f80897fa700 1 -- 172.21.15.129:0/507189561 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f807c01aa30 con 0x7f808c104b30 2024-03-11T20:46:05.474 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.703+0000 7f80897fa700 1 -- 172.21.15.129:0/507189561 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f807c024960 con 0x7f808c104b30 2024-03-11T20:46:05.474 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.703+0000 7f809236d700 1 -- 172.21.15.129:0/507189561 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f808c19acb0 con 0x7f808c104b30 2024-03-11T20:46:05.474 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.704+0000 7f80897fa700 1 -- 172.21.15.129:0/507189561 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7f807c022020 con 0x7f808c104b30 2024-03-11T20:46:05.474 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.704+0000 7f809236d700 1 -- 172.21.15.129:0/507189561 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f808c063410 con 0x7f808c104b30 2024-03-11T20:46:05.474 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.704+0000 7f80897fa700 1 --2- 172.21.15.129:0/507189561 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f807403afc0 0x7f807403d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:05.474 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.705+0000 7f80897fa700 1 -- 172.21.15.129:0/507189561 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f807c04d6d0 con 0x7f808c104b30 2024-03-11T20:46:05.474 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.705+0000 7f808b7fe700 1 --2- 172.21.15.129:0/507189561 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f807403afc0 0x7f807403d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:05.474 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.705+0000 7f808b7fe700 1 --2- 172.21.15.129:0/507189561 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f807403afc0 0x7f807403d480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8080006fd0 tx=0x7f8080006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:05.474 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.710+0000 7f80897fa700 1 -- 172.21.15.129:0/507189561 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f807c00b3e0 con 0x7f808c104b30 2024-03-11T20:46:05.474 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.729+0000 7f80897fa700 1 -- 172.21.15.129:0/507189561 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f807c02dc90 con 0x7f808c104b30 2024-03-11T20:46:05.474 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.924+0000 7f809236d700 1 -- 172.21.15.129:0/507189561 --> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f808c062220 con 0x7f807403afc0 2024-03-11T20:46:05.474 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.931+0000 7f80897fa700 1 -- 172.21.15.129:0/507189561 <== mgr.14120 v2:172.21.15.129:6800/521270824 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f808c062220 con 0x7f807403afc0 2024-03-11T20:46:05.475 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.933+0000 7f809236d700 1 -- 172.21.15.129:0/507189561 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f807403afc0 msgr2=0x7f807403d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:05.475 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.933+0000 7f809236d700 1 --2- 172.21.15.129:0/507189561 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f807403afc0 0x7f807403d480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8080006fd0 tx=0x7f8080006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:05.475 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.933+0000 7f809236d700 1 -- 172.21.15.129:0/507189561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f808c104b30 msgr2=0x7f808c1977f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:05.475 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.933+0000 7f809236d700 1 --2- 172.21.15.129:0/507189561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f808c104b30 0x7f808c1977f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f807c003e40 tx=0x7f807c0119c0 comp rx=0 tx=0).stop 2024-03-11T20:46:05.475 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.934+0000 7f809236d700 1 -- 172.21.15.129:0/507189561 shutdown_connections 2024-03-11T20:46:05.475 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.934+0000 7f809236d700 1 --2- 172.21.15.129:0/507189561 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f807403afc0 0x7f807403d480 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:05.475 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.934+0000 7f809236d700 1 --2- 172.21.15.129:0/507189561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f808c104b30 0x7f808c1977f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:05.475 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.934+0000 7f809236d700 1 -- 172.21.15.129:0/507189561 >> 172.21.15.129:0/507189561 conn(0x7f808c1000b0 msgr2=0x7f808c18de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:05.475 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.934+0000 7f809236d700 1 -- 172.21.15.129:0/507189561 shutdown_connections 2024-03-11T20:46:05.475 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:04.934+0000 7f809236d700 1 -- 172.21.15.129:0/507189561 wait complete. 2024-03-11T20:46:05.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:05 smithi129 ceph-mon[30184]: mgrmap e8: y(active, since 2s) 2024-03-11T20:46:05.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:05 smithi129 ceph-mon[30184]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:46:05.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:05 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' 2024-03-11T20:46:05.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:05 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:06.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-11T20:46:06.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.947+0000 7f4bbac5d700 1 Processor -- start 2024-03-11T20:46:06.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.947+0000 7f4bbac5d700 1 -- start start 2024-03-11T20:46:06.727 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.947+0000 7f4bbac5d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb4105c60 0x7f4bb4106080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:06.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.947+0000 7f4bbac5d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb4106650 con 0x7f4bb4105c60 2024-03-11T20:46:06.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.947+0000 7f4bb89f9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb4105c60 0x7f4bb4106080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:06.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.947+0000 7f4bb89f9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb4105c60 0x7f4bb4106080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39332/0 (socket says 172.21.15.129:39332) 2024-03-11T20:46:06.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.947+0000 7f4bb89f9700 1 -- 172.21.15.129:0/3909447793 learned_addr learned my addr 172.21.15.129:0/3909447793 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:06.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.947+0000 7f4bb89f9700 1 -- 172.21.15.129:0/3909447793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb4106e60 con 0x7f4bb4105c60 2024-03-11T20:46:06.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.948+0000 7f4bb89f9700 1 --2- 172.21.15.129:0/3909447793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb4105c60 0x7f4bb4106080 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80098a0 tx=0x7f4ba8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19ae244f8922ed9 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:06.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.948+0000 7f4bb37fe700 1 -- 172.21.15.129:0/3909447793 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ba8004160 con 0x7f4bb4105c60 2024-03-11T20:46:06.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.948+0000 7f4bb37fe700 1 -- 172.21.15.129:0/3909447793 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f4ba8004300 con 0x7f4bb4105c60 2024-03-11T20:46:06.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.948+0000 7f4bb37fe700 1 -- 172.21.15.129:0/3909447793 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ba8011940 con 0x7f4bb4105c60 2024-03-11T20:46:06.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.948+0000 7f4bbac5d700 1 -- 172.21.15.129:0/3909447793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb4105c60 msgr2=0x7f4bb4106080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:06.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.948+0000 7f4bbac5d700 1 --2- 172.21.15.129:0/3909447793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb4105c60 0x7f4bb4106080 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80098a0 tx=0x7f4ba8009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:06.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.948+0000 7f4bbac5d700 1 -- 172.21.15.129:0/3909447793 shutdown_connections 2024-03-11T20:46:06.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.948+0000 7f4bbac5d700 1 --2- 172.21.15.129:0/3909447793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb4105c60 0x7f4bb4106080 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:06.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.948+0000 7f4bbac5d700 1 -- 172.21.15.129:0/3909447793 >> 172.21.15.129:0/3909447793 conn(0x7f4bb41011c0 msgr2=0x7f4bb4103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:06.728 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.949+0000 7f4bbac5d700 1 -- 172.21.15.129:0/3909447793 shutdown_connections 2024-03-11T20:46:06.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.949+0000 7f4bbac5d700 1 -- 172.21.15.129:0/3909447793 wait complete. 2024-03-11T20:46:06.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.949+0000 7f4bbac5d700 1 Processor -- start 2024-03-11T20:46:06.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.949+0000 7f4bbac5d700 1 -- start start 2024-03-11T20:46:06.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.949+0000 7f4bbac5d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb4105c60 0x7f4bb407b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:06.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.949+0000 7f4bbac5d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb4107cf0 con 0x7f4bb4105c60 2024-03-11T20:46:06.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.949+0000 7f4bb89f9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb4105c60 0x7f4bb407b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:06.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.949+0000 7f4bb89f9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb4105c60 0x7f4bb407b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39348/0 (socket says 172.21.15.129:39348) 2024-03-11T20:46:06.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.949+0000 7f4bb89f9700 1 -- 172.21.15.129:0/4225411588 learned_addr learned my addr 172.21.15.129:0/4225411588 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:06.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.950+0000 7f4bb89f9700 1 -- 172.21.15.129:0/4225411588 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb407b930 con 0x7f4bb4105c60 2024-03-11T20:46:06.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.950+0000 7f4bb89f9700 1 --2- 172.21.15.129:0/4225411588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb4105c60 0x7f4bb407b3f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8011910 tx=0x7f4ba80119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:06.729 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.950+0000 7f4bb1ffb700 1 -- 172.21.15.129:0/4225411588 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ba8003b20 con 0x7f4bb4105c60 2024-03-11T20:46:06.730 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.950+0000 7f4bb1ffb700 1 -- 172.21.15.129:0/4225411588 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f4ba801a890 con 0x7f4bb4105c60 2024-03-11T20:46:06.730 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.950+0000 7f4bbac5d700 1 -- 172.21.15.129:0/4225411588 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb4079b20 con 0x7f4bb4105c60 2024-03-11T20:46:06.730 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.950+0000 7f4bb1ffb700 1 -- 172.21.15.129:0/4225411588 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ba80197f0 con 0x7f4bb4105c60 2024-03-11T20:46:06.730 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.950+0000 7f4bbac5d700 1 -- 172.21.15.129:0/4225411588 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb407a060 con 0x7f4bb4105c60 2024-03-11T20:46:06.730 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.951+0000 7f4bb1ffb700 1 -- 172.21.15.129:0/4225411588 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f4ba8022020 con 0x7f4bb4105c60 2024-03-11T20:46:06.730 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.951+0000 7f4bbac5d700 1 -- 172.21.15.129:0/4225411588 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb4063410 con 0x7f4bb4105c60 2024-03-11T20:46:06.730 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.951+0000 7f4bb1ffb700 1 --2- 172.21.15.129:0/4225411588 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f4ba4038f70 0x7f4ba403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:06.730 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.951+0000 7f4bb1ffb700 1 -- 172.21.15.129:0/4225411588 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4ba804cfe0 con 0x7f4bb4105c60 2024-03-11T20:46:06.730 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.951+0000 7f4bb3fff700 1 --2- 172.21.15.129:0/4225411588 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f4ba4038f70 0x7f4ba403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:06.730 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.951+0000 7f4bb3fff700 1 --2- 172.21.15.129:0/4225411588 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f4ba4038f70 0x7f4ba403b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0006fd0 tx=0x7f4ba0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:06.730 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:05.956+0000 7f4bb1ffb700 1 -- 172.21.15.129:0/4225411588 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f4ba801ad90 con 0x7f4bb4105c60 2024-03-11T20:46:06.730 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:06.152+0000 7f4bbac5d700 1 -- 172.21.15.129:0/4225411588 --> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f4bb407a850 con 0x7f4ba4038f70 2024-03-11T20:46:06.730 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:06.153+0000 7f4bb1ffb700 1 -- 172.21.15.129:0/4225411588 <== mgr.14120 v2:172.21.15.129:6800/521270824 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f4bb407a850 con 0x7f4ba4038f70 2024-03-11T20:46:06.730 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:06.155+0000 7f4bbac5d700 1 -- 172.21.15.129:0/4225411588 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f4ba4038f70 msgr2=0x7f4ba403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:06.731 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:06.155+0000 7f4bbac5d700 1 --2- 172.21.15.129:0/4225411588 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f4ba4038f70 0x7f4ba403b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0006fd0 tx=0x7f4ba0006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:06.731 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:06.155+0000 7f4bbac5d700 1 -- 172.21.15.129:0/4225411588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb4105c60 msgr2=0x7f4bb407b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:06.731 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:06.155+0000 7f4bbac5d700 1 --2- 172.21.15.129:0/4225411588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb4105c60 0x7f4bb407b3f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8011910 tx=0x7f4ba80119f0 comp rx=0 tx=0).stop 2024-03-11T20:46:06.731 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:06.155+0000 7f4bbac5d700 1 -- 172.21.15.129:0/4225411588 shutdown_connections 2024-03-11T20:46:06.731 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:06.155+0000 7f4bbac5d700 1 --2- 172.21.15.129:0/4225411588 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f4ba4038f70 0x7f4ba403b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:06.731 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:06.155+0000 7f4bbac5d700 1 --2- 172.21.15.129:0/4225411588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb4105c60 0x7f4bb407b3f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:06.731 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:06.155+0000 7f4bbac5d700 1 -- 172.21.15.129:0/4225411588 >> 172.21.15.129:0/4225411588 conn(0x7f4bb41011c0 msgr2=0x7f4bb4101e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:06.731 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:06.155+0000 7f4bbac5d700 1 -- 172.21.15.129:0/4225411588 shutdown_connections 2024-03-11T20:46:06.731 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:06.155+0000 7f4bbac5d700 1 -- 172.21.15.129:0/4225411588 wait complete. 2024-03-11T20:46:06.731 INFO:teuthology.orchestra.run.smithi129.stdout:Generating ssh key... 2024-03-11T20:46:07.227 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:06 smithi129 ceph-mon[30184]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:46:08.108 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: Generating public/private rsa key pair. 2024-03-11T20:46:08.109 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: Your identification has been saved in /tmp/tmpj1sidaxx/key. 2024-03-11T20:46:08.109 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: Your public key has been saved in /tmp/tmpj1sidaxx/key.pub. 2024-03-11T20:46:08.109 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: The key fingerprint is: 2024-03-11T20:46:08.109 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: SHA256:jS9rEWJTEfSR7OaKW/wV+Iq2bRLg6+fXBwf4gx4Seas ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:46:08.109 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: The key's randomart image is: 2024-03-11T20:46:08.109 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: +---[RSA 3072]----+ 2024-03-11T20:46:08.109 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: | .++.. | 2024-03-11T20:46:08.109 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: | ..o. | 2024-03-11T20:46:08.109 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: | . o.. | 2024-03-11T20:46:08.109 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: | = +o=.. | 2024-03-11T20:46:08.109 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: | o +S*o+.. | 2024-03-11T20:46:08.109 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: | ..+.=.+.. | 2024-03-11T20:46:08.109 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: | o+B.oo+ | 2024-03-11T20:46:08.110 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: | o.E*+o. . | 2024-03-11T20:46:08.110 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: | .o*==+ . | 2024-03-11T20:46:08.110 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: +----[SHA256]-----+ 2024-03-11T20:46:08.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.169+0000 7fb1d18bb700 1 Processor -- start 2024-03-11T20:46:08.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.169+0000 7fb1d18bb700 1 -- start start 2024-03-11T20:46:08.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.169+0000 7fb1d18bb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1cc1066e0 0x7fb1cc108ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:08.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.169+0000 7fb1d18bb700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc079730 con 0x7fb1cc1066e0 2024-03-11T20:46:08.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.169+0000 7fb1caffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1cc1066e0 0x7fb1cc108ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:08.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.169+0000 7fb1caffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1cc1066e0 0x7fb1cc108ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39364/0 (socket says 172.21.15.129:39364) 2024-03-11T20:46:08.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.169+0000 7fb1caffd700 1 -- 172.21.15.129:0/618701889 learned_addr learned my addr 172.21.15.129:0/618701889 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:08.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.170+0000 7fb1caffd700 1 -- 172.21.15.129:0/618701889 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1cc109010 con 0x7fb1cc1066e0 2024-03-11T20:46:08.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.170+0000 7fb1caffd700 1 --2- 172.21.15.129:0/618701889 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1cc1066e0 0x7fb1cc108ad0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0098a0 tx=0x7fb1bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47cb1c27b05b6a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:08.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.170+0000 7fb1c9ffb700 1 -- 172.21.15.129:0/618701889 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1bc004160 con 0x7fb1cc1066e0 2024-03-11T20:46:08.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.170+0000 7fb1c9ffb700 1 -- 172.21.15.129:0/618701889 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fb1bc004300 con 0x7fb1cc1066e0 2024-03-11T20:46:08.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.170+0000 7fb1c9ffb700 1 -- 172.21.15.129:0/618701889 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1bc011940 con 0x7fb1cc1066e0 2024-03-11T20:46:08.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.170+0000 7fb1d18bb700 1 -- 172.21.15.129:0/618701889 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1cc1066e0 msgr2=0x7fb1cc108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:08.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.170+0000 7fb1d18bb700 1 --2- 172.21.15.129:0/618701889 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1cc1066e0 0x7fb1cc108ad0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0098a0 tx=0x7fb1bc009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:08.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.170+0000 7fb1d18bb700 1 -- 172.21.15.129:0/618701889 shutdown_connections 2024-03-11T20:46:08.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.170+0000 7fb1d18bb700 1 --2- 172.21.15.129:0/618701889 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1cc1066e0 0x7fb1cc108ad0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:08.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.170+0000 7fb1d18bb700 1 -- 172.21.15.129:0/618701889 >> 172.21.15.129:0/618701889 conn(0x7fb1cc1000f0 msgr2=0x7fb1cc102550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:08.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.171+0000 7fb1d18bb700 1 -- 172.21.15.129:0/618701889 shutdown_connections 2024-03-11T20:46:08.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.171+0000 7fb1d18bb700 1 -- 172.21.15.129:0/618701889 wait complete. 2024-03-11T20:46:08.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.171+0000 7fb1d18bb700 1 Processor -- start 2024-03-11T20:46:08.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.171+0000 7fb1d18bb700 1 -- start start 2024-03-11T20:46:08.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.171+0000 7fb1d18bb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1cc1066e0 0x7fb1cc1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:08.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.171+0000 7fb1d18bb700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc109e40 con 0x7fb1cc1066e0 2024-03-11T20:46:08.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.171+0000 7fb1caffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1cc1066e0 0x7fb1cc1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:08.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.171+0000 7fb1caffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1cc1066e0 0x7fb1cc1999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39374/0 (socket says 172.21.15.129:39374) 2024-03-11T20:46:08.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.172+0000 7fb1caffd700 1 -- 172.21.15.129:0/912498801 learned_addr learned my addr 172.21.15.129:0/912498801 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:08.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.172+0000 7fb1caffd700 1 -- 172.21.15.129:0/912498801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1cc199f00 con 0x7fb1cc1066e0 2024-03-11T20:46:08.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.172+0000 7fb1caffd700 1 --2- 172.21.15.129:0/912498801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1cc1066e0 0x7fb1cc1999c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc009870 tx=0x7fb1bc0119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:08.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.172+0000 7fb1d08b9700 1 -- 172.21.15.129:0/912498801 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1bc01a430 con 0x7fb1cc1066e0 2024-03-11T20:46:08.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.172+0000 7fb1d18bb700 1 -- 172.21.15.129:0/912498801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1cc19a190 con 0x7fb1cc1066e0 2024-03-11T20:46:08.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.172+0000 7fb1d08b9700 1 -- 172.21.15.129:0/912498801 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fb1bc01aa30 con 0x7fb1cc1066e0 2024-03-11T20:46:08.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.172+0000 7fb1d08b9700 1 -- 172.21.15.129:0/912498801 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1bc0248d0 con 0x7fb1cc1066e0 2024-03-11T20:46:08.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.172+0000 7fb1d18bb700 1 -- 172.21.15.129:0/912498801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1cc19ce80 con 0x7fb1cc1066e0 2024-03-11T20:46:08.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.173+0000 7fb1d08b9700 1 -- 172.21.15.129:0/912498801 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fb1bc022020 con 0x7fb1cc1066e0 2024-03-11T20:46:08.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.173+0000 7fb1d18bb700 1 -- 172.21.15.129:0/912498801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1cc063410 con 0x7fb1cc1066e0 2024-03-11T20:46:08.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.173+0000 7fb1d08b9700 1 --2- 172.21.15.129:0/912498801 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fb1b4038ed0 0x7fb1b403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:08.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.173+0000 7fb1d08b9700 1 -- 172.21.15.129:0/912498801 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb1bc04d380 con 0x7fb1cc1066e0 2024-03-11T20:46:08.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.173+0000 7fb1ca7fc700 1 --2- 172.21.15.129:0/912498801 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fb1b4038ed0 0x7fb1b403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:08.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.173+0000 7fb1ca7fc700 1 --2- 172.21.15.129:0/912498801 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fb1b4038ed0 0x7fb1b403b390 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0006fd0 tx=0x7fb1c0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:08.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.178+0000 7fb1d08b9700 1 -- 172.21.15.129:0/912498801 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fb1bc019c70 con 0x7fb1cc1066e0 2024-03-11T20:46:08.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.373+0000 7fb1d18bb700 1 -- 172.21.15.129:0/912498801 --> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1cc1012f0 con 0x7fb1b4038ed0 2024-03-11T20:46:08.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.785+0000 7fb1d08b9700 1 -- 172.21.15.129:0/912498801 <== mgr.14120 v2:172.21.15.129:6800/521270824 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb1cc1012f0 con 0x7fb1b4038ed0 2024-03-11T20:46:08.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.787+0000 7fb1d18bb700 1 -- 172.21.15.129:0/912498801 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fb1b4038ed0 msgr2=0x7fb1b403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:08.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.787+0000 7fb1d18bb700 1 --2- 172.21.15.129:0/912498801 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fb1b4038ed0 0x7fb1b403b390 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0006fd0 tx=0x7fb1c0006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:08.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.788+0000 7fb1d18bb700 1 -- 172.21.15.129:0/912498801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1cc1066e0 msgr2=0x7fb1cc1999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:08.387 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.788+0000 7fb1d18bb700 1 --2- 172.21.15.129:0/912498801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1cc1066e0 0x7fb1cc1999c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc009870 tx=0x7fb1bc0119c0 comp rx=0 tx=0).stop 2024-03-11T20:46:08.387 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.788+0000 7fb1d18bb700 1 -- 172.21.15.129:0/912498801 shutdown_connections 2024-03-11T20:46:08.387 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.788+0000 7fb1d18bb700 1 --2- 172.21.15.129:0/912498801 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fb1b4038ed0 0x7fb1b403b390 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:08.387 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.788+0000 7fb1d18bb700 1 --2- 172.21.15.129:0/912498801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1cc1066e0 0x7fb1cc1999c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:08.387 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.788+0000 7fb1d18bb700 1 -- 172.21.15.129:0/912498801 >> 172.21.15.129:0/912498801 conn(0x7fb1cc1000f0 msgr2=0x7fb1cc100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:08.387 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.788+0000 7fb1d18bb700 1 -- 172.21.15.129:0/912498801 shutdown_connections 2024-03-11T20:46:08.387 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:07.788+0000 7fb1d18bb700 1 -- 172.21.15.129:0/912498801 wait complete. 2024-03-11T20:46:09.072 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:08 smithi129 ceph-mon[30184]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:46:09.072 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:08 smithi129 ceph-mon[30184]: Generating ssh key... 2024-03-11T20:46:09.072 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:08 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' 2024-03-11T20:46:09.072 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:08 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' 2024-03-11T20:46:09.599 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2skFy55BXJHmBwHv2vClbQr5gHVeXZUepErQ2bx63MX+w3TwLc5KHSWM+Zr67WfhAeH/ToOJaeP6fY7yvTn5HWv4vEXtoqreW4wz0XD4JCCcjFTlR25DCADV68prbishdZOnQEwnQSOPFN6wj6DAtOqtlcZGLYz7sI/QDRejQUEGyaCEb1ZNTH3P7KBYLFUHnZ6pO63PietHinDhyaq/kCI58N3ND9rEiIyeqfjHG8q8bIapTyVMig/AmAIFjRHmBhlzdl+Y6cAKAmLa3r6ek5yp19ufDv8u378Kmki1u1M1N3YYjbKbGvZpwaNEkKkWNB2zalfOyNetDXwxYmaiK6LPhNzVvDRKB1a6sxAApNfUZibiLdmDkvsgh5YtI56HOgXVRSjTJgeexB2+7FXiBmFhx705HJ8APPNS6YkYyXfBRqQIyjVZhfrsdo+qWH7kl/99EEAI53Fcwo4KVhm89PHwAfjsgtLl1WNyoJSgCJThSV7LGpxwn0FL6ytnIqUc= ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:46:09.599 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.874+0000 7f1b5cdef700 1 Processor -- start 2024-03-11T20:46:09.599 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.874+0000 7f1b5cdef700 1 -- start start 2024-03-11T20:46:09.599 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.875+0000 7f1b5cdef700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b58106720 0x7f1b58108b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:09.599 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.875+0000 7f1b5cdef700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b580796f0 con 0x7f1b58106720 2024-03-11T20:46:09.600 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.875+0000 7f1b5659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b58106720 0x7f1b58108b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:09.600 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.875+0000 7f1b5659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b58106720 0x7f1b58108b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39384/0 (socket says 172.21.15.129:39384) 2024-03-11T20:46:09.600 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.875+0000 7f1b5659c700 1 -- 172.21.15.129:0/3426333668 learned_addr learned my addr 172.21.15.129:0/3426333668 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:09.600 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.875+0000 7f1b5659c700 1 -- 172.21.15.129:0/3426333668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b58109050 con 0x7f1b58106720 2024-03-11T20:46:09.600 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.875+0000 7f1b5659c700 1 --2- 172.21.15.129:0/3426333668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b58106720 0x7f1b58108b10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0098a0 tx=0x7f1b4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d132f0a989e4799 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:09.600 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.876+0000 7f1b5559a700 1 -- 172.21.15.129:0/3426333668 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b4c004160 con 0x7f1b58106720 2024-03-11T20:46:09.600 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.876+0000 7f1b5559a700 1 -- 172.21.15.129:0/3426333668 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f1b4c004300 con 0x7f1b58106720 2024-03-11T20:46:09.600 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.876+0000 7f1b5559a700 1 -- 172.21.15.129:0/3426333668 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b4c011940 con 0x7f1b58106720 2024-03-11T20:46:09.600 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.876+0000 7f1b5cdef700 1 -- 172.21.15.129:0/3426333668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b58106720 msgr2=0x7f1b58108b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:09.600 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.876+0000 7f1b5cdef700 1 --2- 172.21.15.129:0/3426333668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b58106720 0x7f1b58108b10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0098a0 tx=0x7f1b4c009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:09.601 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.876+0000 7f1b5cdef700 1 -- 172.21.15.129:0/3426333668 shutdown_connections 2024-03-11T20:46:09.601 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.876+0000 7f1b5cdef700 1 --2- 172.21.15.129:0/3426333668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b58106720 0x7f1b58108b10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:09.601 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.876+0000 7f1b5cdef700 1 -- 172.21.15.129:0/3426333668 >> 172.21.15.129:0/3426333668 conn(0x7f1b581000f0 msgr2=0x7f1b58102550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:09.601 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.876+0000 7f1b5cdef700 1 -- 172.21.15.129:0/3426333668 shutdown_connections 2024-03-11T20:46:09.601 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.876+0000 7f1b5cdef700 1 -- 172.21.15.129:0/3426333668 wait complete. 2024-03-11T20:46:09.601 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.877+0000 7f1b5cdef700 1 Processor -- start 2024-03-11T20:46:09.601 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.877+0000 7f1b5cdef700 1 -- start start 2024-03-11T20:46:09.601 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.877+0000 7f1b5cdef700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b58106720 0x7f1b5819bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:09.601 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.877+0000 7f1b5cdef700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b58109e70 con 0x7f1b58106720 2024-03-11T20:46:09.601 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.877+0000 7f1b5659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b58106720 0x7f1b5819bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:09.601 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.877+0000 7f1b5659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b58106720 0x7f1b5819bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39388/0 (socket says 172.21.15.129:39388) 2024-03-11T20:46:09.601 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.877+0000 7f1b5659c700 1 -- 172.21.15.129:0/4063522220 learned_addr learned my addr 172.21.15.129:0/4063522220 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:09.601 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.878+0000 7f1b5659c700 1 -- 172.21.15.129:0/4063522220 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5819c120 con 0x7f1b58106720 2024-03-11T20:46:09.602 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.878+0000 7f1b5659c700 1 --2- 172.21.15.129:0/4063522220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b58106720 0x7f1b5819bbe0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c003e40 tx=0x7f1b4c0119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:09.602 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.878+0000 7f1b477fe700 1 -- 172.21.15.129:0/4063522220 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b4c003b20 con 0x7f1b58106720 2024-03-11T20:46:09.602 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.878+0000 7f1b477fe700 1 -- 172.21.15.129:0/4063522220 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f1b4c01a890 con 0x7f1b58106720 2024-03-11T20:46:09.602 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.878+0000 7f1b5cdef700 1 -- 172.21.15.129:0/4063522220 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5819c3b0 con 0x7f1b58106720 2024-03-11T20:46:09.602 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.878+0000 7f1b477fe700 1 -- 172.21.15.129:0/4063522220 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b4c024890 con 0x7f1b58106720 2024-03-11T20:46:09.602 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.878+0000 7f1b5cdef700 1 -- 172.21.15.129:0/4063522220 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5819f0a0 con 0x7f1b58106720 2024-03-11T20:46:09.602 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.879+0000 7f1b477fe700 1 -- 172.21.15.129:0/4063522220 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f1b4c022020 con 0x7f1b58106720 2024-03-11T20:46:09.602 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.879+0000 7f1b5cdef700 1 -- 172.21.15.129:0/4063522220 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b58063410 con 0x7f1b58106720 2024-03-11T20:46:09.602 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.879+0000 7f1b477fe700 1 --2- 172.21.15.129:0/4063522220 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f1b40038f20 0x7f1b4003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:09.602 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.879+0000 7f1b477fe700 1 -- 172.21.15.129:0/4063522220 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1b4c04cff0 con 0x7f1b58106720 2024-03-11T20:46:09.602 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.879+0000 7f1b55d9b700 1 --2- 172.21.15.129:0/4063522220 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f1b40038f20 0x7f1b4003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:09.602 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.880+0000 7f1b55d9b700 1 --2- 172.21.15.129:0/4063522220 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f1b40038f20 0x7f1b4003b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1b48006fd0 tx=0x7f1b48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:09.603 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:08.884+0000 7f1b477fe700 1 -- 172.21.15.129:0/4063522220 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f1b4c00b3e0 con 0x7f1b58106720 2024-03-11T20:46:09.603 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:09.077+0000 7f1b5cdef700 1 -- 172.21.15.129:0/4063522220 --> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b58101270 con 0x7f1b40038f20 2024-03-11T20:46:09.603 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:09.078+0000 7f1b477fe700 1 -- 172.21.15.129:0/4063522220 <== mgr.14120 v2:172.21.15.129:6800/521270824 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f1b58101270 con 0x7f1b40038f20 2024-03-11T20:46:09.603 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:09.080+0000 7f1b5cdef700 1 -- 172.21.15.129:0/4063522220 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f1b40038f20 msgr2=0x7f1b4003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:09.603 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:09.080+0000 7f1b5cdef700 1 --2- 172.21.15.129:0/4063522220 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f1b40038f20 0x7f1b4003b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1b48006fd0 tx=0x7f1b48006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:09.603 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:09.080+0000 7f1b5cdef700 1 -- 172.21.15.129:0/4063522220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b58106720 msgr2=0x7f1b5819bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:09.603 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:09.080+0000 7f1b5cdef700 1 --2- 172.21.15.129:0/4063522220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b58106720 0x7f1b5819bbe0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c003e40 tx=0x7f1b4c0119c0 comp rx=0 tx=0).stop 2024-03-11T20:46:09.603 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:09.080+0000 7f1b5cdef700 1 -- 172.21.15.129:0/4063522220 shutdown_connections 2024-03-11T20:46:09.603 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:09.080+0000 7f1b5cdef700 1 --2- 172.21.15.129:0/4063522220 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f1b40038f20 0x7f1b4003b3e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:09.603 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:09.080+0000 7f1b5cdef700 1 --2- 172.21.15.129:0/4063522220 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b58106720 0x7f1b5819bbe0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:09.603 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:09.080+0000 7f1b5cdef700 1 -- 172.21.15.129:0/4063522220 >> 172.21.15.129:0/4063522220 conn(0x7f1b581000f0 msgr2=0x7f1b58100b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:09.603 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:09.080+0000 7f1b5cdef700 1 -- 172.21.15.129:0/4063522220 shutdown_connections 2024-03-11T20:46:09.603 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:09.080+0000 7f1b5cdef700 1 -- 172.21.15.129:0/4063522220 wait complete. 2024-03-11T20:46:09.603 INFO:teuthology.orchestra.run.smithi129.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T20:46:09.604 INFO:teuthology.orchestra.run.smithi129.stdout:Adding key to root@localhost authorized_keys... 2024-03-11T20:46:09.604 INFO:teuthology.orchestra.run.smithi129.stdout:Adding host smithi129... 2024-03-11T20:46:10.227 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:09 smithi129 ceph-mon[30184]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:46:11.152 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:10 smithi129 ceph-mon[30184]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi129", "addr": "172.21.15.129", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:46:12.227 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:11 smithi129 ceph-mon[30184]: Deploying cephadm binary to smithi129 2024-03-11T20:46:15.106 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout Added host 'smithi129' with addr '172.21.15.129' 2024-03-11T20:46:15.106 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.051+0000 7f7399636700 1 Processor -- start 2024-03-11T20:46:15.106 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.052+0000 7f7399636700 1 -- start start 2024-03-11T20:46:15.106 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.052+0000 7f7399636700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7394105c70 0x7f7394106090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:15.106 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.052+0000 7f7399636700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7394106660 con 0x7f7394105c70 2024-03-11T20:46:15.106 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.052+0000 7f7392ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7394105c70 0x7f7394106090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:15.106 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.052+0000 7f7392ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7394105c70 0x7f7394106090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39390/0 (socket says 172.21.15.129:39390) 2024-03-11T20:46:15.106 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.052+0000 7f7392ffd700 1 -- 172.21.15.129:0/2525201077 learned_addr learned my addr 172.21.15.129:0/2525201077 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:15.107 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.052+0000 7f7392ffd700 1 -- 172.21.15.129:0/2525201077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7394106e70 con 0x7f7394105c70 2024-03-11T20:46:15.107 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.052+0000 7f7392ffd700 1 --2- 172.21.15.129:0/2525201077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7394105c70 0x7f7394106090 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f73840098a0 tx=0x7f7384009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486331ecaf39c4b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:15.107 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.052+0000 7f7391ffb700 1 -- 172.21.15.129:0/2525201077 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7384004160 con 0x7f7394105c70 2024-03-11T20:46:15.107 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.052+0000 7f7391ffb700 1 -- 172.21.15.129:0/2525201077 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f7384004300 con 0x7f7394105c70 2024-03-11T20:46:15.107 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.052+0000 7f7391ffb700 1 -- 172.21.15.129:0/2525201077 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7384011940 con 0x7f7394105c70 2024-03-11T20:46:15.107 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.053+0000 7f7399636700 1 -- 172.21.15.129:0/2525201077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7394105c70 msgr2=0x7f7394106090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:15.107 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.053+0000 7f7399636700 1 --2- 172.21.15.129:0/2525201077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7394105c70 0x7f7394106090 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f73840098a0 tx=0x7f7384009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:15.107 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.053+0000 7f7399636700 1 -- 172.21.15.129:0/2525201077 shutdown_connections 2024-03-11T20:46:15.107 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.053+0000 7f7399636700 1 --2- 172.21.15.129:0/2525201077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7394105c70 0x7f7394106090 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:15.107 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.053+0000 7f7399636700 1 -- 172.21.15.129:0/2525201077 >> 172.21.15.129:0/2525201077 conn(0x7f73941011f0 msgr2=0x7f7394103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:15.107 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.053+0000 7f7399636700 1 -- 172.21.15.129:0/2525201077 shutdown_connections 2024-03-11T20:46:15.107 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.053+0000 7f7399636700 1 -- 172.21.15.129:0/2525201077 wait complete. 2024-03-11T20:46:15.107 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.053+0000 7f7399636700 1 Processor -- start 2024-03-11T20:46:15.108 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.054+0000 7f7399636700 1 -- start start 2024-03-11T20:46:15.108 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.054+0000 7f7399636700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7394105c70 0x7f739407b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:15.108 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.054+0000 7f7399636700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7394107d00 con 0x7f7394105c70 2024-03-11T20:46:15.108 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.054+0000 7f7392ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7394105c70 0x7f739407b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:15.108 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.054+0000 7f7392ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7394105c70 0x7f739407b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39398/0 (socket says 172.21.15.129:39398) 2024-03-11T20:46:15.108 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.054+0000 7f7392ffd700 1 -- 172.21.15.129:0/807136103 learned_addr learned my addr 172.21.15.129:0/807136103 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:15.108 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.054+0000 7f7392ffd700 1 -- 172.21.15.129:0/807136103 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739407b930 con 0x7f7394105c70 2024-03-11T20:46:15.108 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.054+0000 7f7392ffd700 1 --2- 172.21.15.129:0/807136103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7394105c70 0x7f739407b3f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7384009fb0 tx=0x7f73840119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:15.108 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.054+0000 7f737bfff700 1 -- 172.21.15.129:0/807136103 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f738401a430 con 0x7f7394105c70 2024-03-11T20:46:15.108 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.054+0000 7f737bfff700 1 -- 172.21.15.129:0/807136103 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f738401aa30 con 0x7f7394105c70 2024-03-11T20:46:15.108 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.054+0000 7f7399636700 1 -- 172.21.15.129:0/807136103 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7394079b20 con 0x7f7394105c70 2024-03-11T20:46:15.108 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.055+0000 7f737bfff700 1 -- 172.21.15.129:0/807136103 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7384024860 con 0x7f7394105c70 2024-03-11T20:46:15.108 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.055+0000 7f7399636700 1 -- 172.21.15.129:0/807136103 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739407a060 con 0x7f7394105c70 2024-03-11T20:46:15.108 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.055+0000 7f737bfff700 1 -- 172.21.15.129:0/807136103 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f7384022020 con 0x7f7394105c70 2024-03-11T20:46:15.109 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.055+0000 7f7399636700 1 -- 172.21.15.129:0/807136103 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7394063410 con 0x7f7394105c70 2024-03-11T20:46:15.109 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.055+0000 7f737bfff700 1 --2- 172.21.15.129:0/807136103 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f737c038f70 0x7f737c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:15.109 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.056+0000 7f737bfff700 1 -- 172.21.15.129:0/807136103 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f738404d140 con 0x7f7394105c70 2024-03-11T20:46:15.109 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.056+0000 7f73927fc700 1 --2- 172.21.15.129:0/807136103 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f737c038f70 0x7f737c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:15.109 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.056+0000 7f73927fc700 1 --2- 172.21.15.129:0/807136103 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f737c038f70 0x7f737c03b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7388006fd0 tx=0x7f7388006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:15.109 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.060+0000 7f737bfff700 1 -- 172.21.15.129:0/807136103 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f738400b3e0 con 0x7f7394105c70 2024-03-11T20:46:15.109 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:10.257+0000 7f7399636700 1 -- 172.21.15.129:0/807136103 --> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi129", "addr": "172.21.15.129", "target": ["mon-mgr", ""]}) v1 -- 0x7f739407a850 con 0x7f737c038f70 2024-03-11T20:46:15.109 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:14.526+0000 7f737bfff700 1 -- 172.21.15.129:0/807136103 <== mgr.14120 v2:172.21.15.129:6800/521270824 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f739407a850 con 0x7f737c038f70 2024-03-11T20:46:15.109 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:14.529+0000 7f7399636700 1 -- 172.21.15.129:0/807136103 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f737c038f70 msgr2=0x7f737c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:15.109 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:14.529+0000 7f7399636700 1 --2- 172.21.15.129:0/807136103 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f737c038f70 0x7f737c03b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7388006fd0 tx=0x7f7388006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:15.109 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:14.529+0000 7f7399636700 1 -- 172.21.15.129:0/807136103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7394105c70 msgr2=0x7f739407b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:15.109 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:14.529+0000 7f7399636700 1 --2- 172.21.15.129:0/807136103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7394105c70 0x7f739407b3f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7384009fb0 tx=0x7f73840119c0 comp rx=0 tx=0).stop 2024-03-11T20:46:15.109 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:14.529+0000 7f7399636700 1 -- 172.21.15.129:0/807136103 shutdown_connections 2024-03-11T20:46:15.110 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:14.529+0000 7f7399636700 1 --2- 172.21.15.129:0/807136103 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f737c038f70 0x7f737c03b430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:15.110 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:14.529+0000 7f7399636700 1 --2- 172.21.15.129:0/807136103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7394105c70 0x7f739407b3f0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7384009fb0 tx=0x7f73840119c0 comp rx=0 tx=0).stop 2024-03-11T20:46:15.110 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:14.529+0000 7f7399636700 1 -- 172.21.15.129:0/807136103 >> 172.21.15.129:0/807136103 conn(0x7f73941011f0 msgr2=0x7f7394101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:15.110 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:14.529+0000 7f7399636700 1 -- 172.21.15.129:0/807136103 shutdown_connections 2024-03-11T20:46:15.110 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:14.529+0000 7f7399636700 1 -- 172.21.15.129:0/807136103 wait complete. 2024-03-11T20:46:15.110 INFO:teuthology.orchestra.run.smithi129.stdout:Deploying unmanaged mon service... 2024-03-11T20:46:15.621 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:15 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' 2024-03-11T20:46:15.621 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:15 smithi129 ceph-mon[30184]: Added host smithi129 2024-03-11T20:46:15.621 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:15 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:16.640 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-11T20:46:16.640 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.626+0000 7f717fca3700 1 Processor -- start 2024-03-11T20:46:16.640 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.626+0000 7f717fca3700 1 -- start start 2024-03-11T20:46:16.641 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.626+0000 7f717fca3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7178106700 0x7f7178108af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:16.641 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.626+0000 7f717fca3700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7178078e40 con 0x7f7178106700 2024-03-11T20:46:16.641 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.627+0000 7f717da3f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7178106700 0x7f7178108af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:16.641 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.627+0000 7f717da3f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7178106700 0x7f7178108af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34958/0 (socket says 172.21.15.129:34958) 2024-03-11T20:46:16.641 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.627+0000 7f717da3f700 1 -- 172.21.15.129:0/1556773515 learned_addr learned my addr 172.21.15.129:0/1556773515 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:16.641 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.627+0000 7f717da3f700 1 -- 172.21.15.129:0/1556773515 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7178109030 con 0x7f7178106700 2024-03-11T20:46:16.641 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.627+0000 7f717da3f700 1 --2- 172.21.15.129:0/1556773515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7178106700 0x7f7178108af0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f71740098a0 tx=0x7f7174009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af620787dc338667 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:16.641 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.627+0000 7f717ca3d700 1 -- 172.21.15.129:0/1556773515 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7174004160 con 0x7f7178106700 2024-03-11T20:46:16.641 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.627+0000 7f717ca3d700 1 -- 172.21.15.129:0/1556773515 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f7174004300 con 0x7f7178106700 2024-03-11T20:46:16.641 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.627+0000 7f717ca3d700 1 -- 172.21.15.129:0/1556773515 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7174011940 con 0x7f7178106700 2024-03-11T20:46:16.641 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.628+0000 7f717fca3700 1 -- 172.21.15.129:0/1556773515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7178106700 msgr2=0x7f7178108af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:16.641 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.628+0000 7f717fca3700 1 --2- 172.21.15.129:0/1556773515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7178106700 0x7f7178108af0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f71740098a0 tx=0x7f7174009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:16.641 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.628+0000 7f717fca3700 1 -- 172.21.15.129:0/1556773515 shutdown_connections 2024-03-11T20:46:16.642 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.628+0000 7f717fca3700 1 --2- 172.21.15.129:0/1556773515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7178106700 0x7f7178108af0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:16.642 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.628+0000 7f717fca3700 1 -- 172.21.15.129:0/1556773515 >> 172.21.15.129:0/1556773515 conn(0x7f71781000d0 msgr2=0x7f7178102510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:16.642 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.628+0000 7f717fca3700 1 -- 172.21.15.129:0/1556773515 shutdown_connections 2024-03-11T20:46:16.642 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.628+0000 7f717fca3700 1 -- 172.21.15.129:0/1556773515 wait complete. 2024-03-11T20:46:16.642 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.629+0000 7f717fca3700 1 Processor -- start 2024-03-11T20:46:16.642 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.629+0000 7f717fca3700 1 -- start start 2024-03-11T20:46:16.642 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.629+0000 7f717fca3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7178106700 0x7f7178199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:16.642 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.629+0000 7f717fca3700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7178109e00 con 0x7f7178106700 2024-03-11T20:46:16.642 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.629+0000 7f717da3f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7178106700 0x7f7178199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:16.642 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.629+0000 7f717da3f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7178106700 0x7f7178199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34966/0 (socket says 172.21.15.129:34966) 2024-03-11T20:46:16.642 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.629+0000 7f717da3f700 1 -- 172.21.15.129:0/3882209961 learned_addr learned my addr 172.21.15.129:0/3882209961 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:16.642 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.629+0000 7f717da3f700 1 -- 172.21.15.129:0/3882209961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7178199f70 con 0x7f7178106700 2024-03-11T20:46:16.642 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.629+0000 7f717da3f700 1 --2- 172.21.15.129:0/3882209961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7178106700 0x7f7178199a30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7174009290 tx=0x7f71740119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:16.643 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.629+0000 7f716effd700 1 -- 172.21.15.129:0/3882209961 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7174003930 con 0x7f7178106700 2024-03-11T20:46:16.643 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.629+0000 7f717fca3700 1 -- 172.21.15.129:0/3882209961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f717819a200 con 0x7f7178106700 2024-03-11T20:46:16.643 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.629+0000 7f716effd700 1 -- 172.21.15.129:0/3882209961 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7f7174003ad0 con 0x7f7178106700 2024-03-11T20:46:16.643 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.629+0000 7f716effd700 1 -- 172.21.15.129:0/3882209961 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7174019720 con 0x7f7178106700 2024-03-11T20:46:16.643 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.629+0000 7f717fca3700 1 -- 172.21.15.129:0/3882209961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f717819cef0 con 0x7f7178106700 2024-03-11T20:46:16.643 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.630+0000 7f716effd700 1 -- 172.21.15.129:0/3882209961 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f7174019a20 con 0x7f7178106700 2024-03-11T20:46:16.643 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.630+0000 7f716effd700 1 --2- 172.21.15.129:0/3882209961 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f7164038ed0 0x7f716403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:16.643 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.630+0000 7f716effd700 1 -- 172.21.15.129:0/3882209961 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f717404d5a0 con 0x7f7178106700 2024-03-11T20:46:16.643 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.630+0000 7f717fca3700 1 -- 172.21.15.129:0/3882209961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7178063410 con 0x7f7178106700 2024-03-11T20:46:16.643 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.630+0000 7f717d23e700 1 --2- 172.21.15.129:0/3882209961 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f7164038ed0 0x7f716403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:16.643 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.631+0000 7f717d23e700 1 --2- 172.21.15.129:0/3882209961 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f7164038ed0 0x7f716403b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7168006fd0 tx=0x7f7168006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:16.643 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.635+0000 7f716effd700 1 -- 172.21.15.129:0/3882209961 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f7174019d60 con 0x7f7178106700 2024-03-11T20:46:16.643 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.843+0000 7f717fca3700 1 -- 172.21.15.129:0/3882209961 --> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f71781013e0 con 0x7f7164038ed0 2024-03-11T20:46:16.643 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.848+0000 7f716effd700 1 -- 172.21.15.129:0/3882209961 <== mgr.14120 v2:172.21.15.129:6800/521270824 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f71781013e0 con 0x7f7164038ed0 2024-03-11T20:46:16.644 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.850+0000 7f717fca3700 1 -- 172.21.15.129:0/3882209961 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f7164038ed0 msgr2=0x7f716403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:16.644 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.850+0000 7f717fca3700 1 --2- 172.21.15.129:0/3882209961 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f7164038ed0 0x7f716403b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7168006fd0 tx=0x7f7168006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:16.644 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.850+0000 7f717fca3700 1 -- 172.21.15.129:0/3882209961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7178106700 msgr2=0x7f7178199a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:16.644 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.850+0000 7f717fca3700 1 --2- 172.21.15.129:0/3882209961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7178106700 0x7f7178199a30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7174009290 tx=0x7f71740119c0 comp rx=0 tx=0).stop 2024-03-11T20:46:16.644 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.850+0000 7f717fca3700 1 -- 172.21.15.129:0/3882209961 shutdown_connections 2024-03-11T20:46:16.644 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.850+0000 7f717fca3700 1 --2- 172.21.15.129:0/3882209961 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f7164038ed0 0x7f716403b390 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:16.644 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.850+0000 7f717fca3700 1 --2- 172.21.15.129:0/3882209961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7178106700 0x7f7178199a30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:16.644 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.850+0000 7f717fca3700 1 -- 172.21.15.129:0/3882209961 >> 172.21.15.129:0/3882209961 conn(0x7f71781000d0 msgr2=0x7f7178100cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:16.644 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.850+0000 7f717fca3700 1 -- 172.21.15.129:0/3882209961 shutdown_connections 2024-03-11T20:46:16.644 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:15.850+0000 7f717fca3700 1 -- 172.21.15.129:0/3882209961 wait complete. 2024-03-11T20:46:16.644 INFO:teuthology.orchestra.run.smithi129.stdout:Deploying unmanaged mgr service... 2024-03-11T20:46:16.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:16 smithi129 ceph-mon[30184]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:46:16.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:16 smithi129 ceph-mon[30184]: Saving service mon spec with placement count:5 2024-03-11T20:46:16.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:16 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' 2024-03-11T20:46:18.401 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-11T20:46:18.401 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.511+0000 7fa4cc99f700 1 Processor -- start 2024-03-11T20:46:18.401 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.511+0000 7fa4cc99f700 1 -- start start 2024-03-11T20:46:18.401 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.512+0000 7fa4cc99f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c8105c50 0x7fa4c8106070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:18.401 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.512+0000 7fa4cc99f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c8106640 con 0x7fa4c8105c50 2024-03-11T20:46:18.401 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.512+0000 7fa4c659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c8105c50 0x7fa4c8106070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:18.401 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.512+0000 7fa4c659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c8105c50 0x7fa4c8106070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34972/0 (socket says 172.21.15.129:34972) 2024-03-11T20:46:18.401 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.512+0000 7fa4c659c700 1 -- 172.21.15.129:0/3607850820 learned_addr learned my addr 172.21.15.129:0/3607850820 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:18.402 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.512+0000 7fa4c659c700 1 -- 172.21.15.129:0/3607850820 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c8106e50 con 0x7fa4c8105c50 2024-03-11T20:46:18.402 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.512+0000 7fa4c659c700 1 --2- 172.21.15.129:0/3607850820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c8105c50 0x7fa4c8106070 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0098a0 tx=0x7fa4bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=565849b5a33e8e21 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:18.402 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.512+0000 7fa4c559a700 1 -- 172.21.15.129:0/3607850820 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa4bc004160 con 0x7fa4c8105c50 2024-03-11T20:46:18.402 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.512+0000 7fa4c559a700 1 -- 172.21.15.129:0/3607850820 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fa4bc004300 con 0x7fa4c8105c50 2024-03-11T20:46:18.402 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.512+0000 7fa4c559a700 1 -- 172.21.15.129:0/3607850820 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa4bc011940 con 0x7fa4c8105c50 2024-03-11T20:46:18.402 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.513+0000 7fa4cc99f700 1 -- 172.21.15.129:0/3607850820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c8105c50 msgr2=0x7fa4c8106070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:18.402 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.513+0000 7fa4cc99f700 1 --2- 172.21.15.129:0/3607850820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c8105c50 0x7fa4c8106070 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0098a0 tx=0x7fa4bc009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:18.402 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.513+0000 7fa4cc99f700 1 -- 172.21.15.129:0/3607850820 shutdown_connections 2024-03-11T20:46:18.402 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.513+0000 7fa4cc99f700 1 --2- 172.21.15.129:0/3607850820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c8105c50 0x7fa4c8106070 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:18.402 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.513+0000 7fa4cc99f700 1 -- 172.21.15.129:0/3607850820 >> 172.21.15.129:0/3607850820 conn(0x7fa4c81011b0 msgr2=0x7fa4c8103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:18.402 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.513+0000 7fa4cc99f700 1 -- 172.21.15.129:0/3607850820 shutdown_connections 2024-03-11T20:46:18.402 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.513+0000 7fa4cc99f700 1 -- 172.21.15.129:0/3607850820 wait complete. 2024-03-11T20:46:18.402 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.514+0000 7fa4cc99f700 1 Processor -- start 2024-03-11T20:46:18.403 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.514+0000 7fa4cc99f700 1 -- start start 2024-03-11T20:46:18.403 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.514+0000 7fa4cc99f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c8105c50 0x7fa4c8199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:18.403 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.514+0000 7fa4cc99f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c8107ce0 con 0x7fa4c8105c50 2024-03-11T20:46:18.403 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.514+0000 7fa4c659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c8105c50 0x7fa4c8199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:18.403 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.514+0000 7fa4c659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c8105c50 0x7fa4c8199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34976/0 (socket says 172.21.15.129:34976) 2024-03-11T20:46:18.403 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.514+0000 7fa4c659c700 1 -- 172.21.15.129:0/2170888781 learned_addr learned my addr 172.21.15.129:0/2170888781 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:18.403 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.514+0000 7fa4c659c700 1 -- 172.21.15.129:0/2170888781 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c8199f40 con 0x7fa4c8105c50 2024-03-11T20:46:18.403 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.514+0000 7fa4c659c700 1 --2- 172.21.15.129:0/2170888781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c8105c50 0x7fa4c8199a00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009fb0 tx=0x7fa4bc0119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:18.403 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.515+0000 7fa4b77fe700 1 -- 172.21.15.129:0/2170888781 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa4bc01a430 con 0x7fa4c8105c50 2024-03-11T20:46:18.403 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.515+0000 7fa4cc99f700 1 -- 172.21.15.129:0/2170888781 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c819a1d0 con 0x7fa4c8105c50 2024-03-11T20:46:18.403 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.515+0000 7fa4b77fe700 1 -- 172.21.15.129:0/2170888781 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1011+0+0 (secure 0 0 0) 0x7fa4bc01aa30 con 0x7fa4c8105c50 2024-03-11T20:46:18.403 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.515+0000 7fa4b77fe700 1 -- 172.21.15.129:0/2170888781 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa4bc024860 con 0x7fa4c8105c50 2024-03-11T20:46:18.403 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.515+0000 7fa4cc99f700 1 -- 172.21.15.129:0/2170888781 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c819cec0 con 0x7fa4c8105c50 2024-03-11T20:46:18.404 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.515+0000 7fa4b77fe700 1 -- 172.21.15.129:0/2170888781 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fa4bc022020 con 0x7fa4c8105c50 2024-03-11T20:46:18.404 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.516+0000 7fa4cc99f700 1 -- 172.21.15.129:0/2170888781 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c8063410 con 0x7fa4c8105c50 2024-03-11T20:46:18.404 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.516+0000 7fa4b77fe700 1 --2- 172.21.15.129:0/2170888781 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fa4b0038f20 0x7fa4b003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:18.404 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.516+0000 7fa4b77fe700 1 -- 172.21.15.129:0/2170888781 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa4bc04cf80 con 0x7fa4c8105c50 2024-03-11T20:46:18.404 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.516+0000 7fa4c5d9b700 1 --2- 172.21.15.129:0/2170888781 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fa4b0038f20 0x7fa4b003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:18.404 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.516+0000 7fa4c5d9b700 1 --2- 172.21.15.129:0/2170888781 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fa4b0038f20 0x7fa4b003b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8006fd0 tx=0x7fa4b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:18.404 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.520+0000 7fa4b77fe700 1 -- 172.21.15.129:0/2170888781 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fa4bc019e10 con 0x7fa4c8105c50 2024-03-11T20:46:18.404 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.732+0000 7fa4cc99f700 1 -- 172.21.15.129:0/2170888781 --> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa4c81024a0 con 0x7fa4b0038f20 2024-03-11T20:46:18.404 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.736+0000 7fa4b77fe700 1 -- 172.21.15.129:0/2170888781 <== mgr.14120 v2:172.21.15.129:6800/521270824 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa4c81024a0 con 0x7fa4b0038f20 2024-03-11T20:46:18.404 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.738+0000 7fa4cc99f700 1 -- 172.21.15.129:0/2170888781 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fa4b0038f20 msgr2=0x7fa4b003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:18.404 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.738+0000 7fa4cc99f700 1 --2- 172.21.15.129:0/2170888781 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fa4b0038f20 0x7fa4b003b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8006fd0 tx=0x7fa4b8006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:18.404 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.738+0000 7fa4cc99f700 1 -- 172.21.15.129:0/2170888781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c8105c50 msgr2=0x7fa4c8199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:18.404 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.738+0000 7fa4cc99f700 1 --2- 172.21.15.129:0/2170888781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c8105c50 0x7fa4c8199a00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009fb0 tx=0x7fa4bc0119c0 comp rx=0 tx=0).stop 2024-03-11T20:46:18.404 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.738+0000 7fa4cc99f700 1 -- 172.21.15.129:0/2170888781 shutdown_connections 2024-03-11T20:46:18.405 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.738+0000 7fa4cc99f700 1 --2- 172.21.15.129:0/2170888781 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fa4b0038f20 0x7fa4b003b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:18.405 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.738+0000 7fa4cc99f700 1 --2- 172.21.15.129:0/2170888781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4c8105c50 0x7fa4c8199a00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:18.405 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.738+0000 7fa4cc99f700 1 -- 172.21.15.129:0/2170888781 >> 172.21.15.129:0/2170888781 conn(0x7fa4c81011b0 msgr2=0x7fa4c8101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:18.405 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.738+0000 7fa4cc99f700 1 -- 172.21.15.129:0/2170888781 shutdown_connections 2024-03-11T20:46:18.405 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:17.738+0000 7fa4cc99f700 1 -- 172.21.15.129:0/2170888781 wait complete. 2024-03-11T20:46:19.083 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:18 smithi129 ceph-mon[30184]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:46:19.083 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:18 smithi129 ceph-mon[30184]: Saving service mgr spec with placement count:2 2024-03-11T20:46:19.083 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:18 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' 2024-03-11T20:46:19.084 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:18 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' 2024-03-11T20:46:19.084 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:18 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' 2024-03-11T20:46:20.019 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.864+0000 7f4d6625e700 1 Processor -- start 2024-03-11T20:46:20.019 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.864+0000 7f4d6625e700 1 -- start start 2024-03-11T20:46:20.019 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.864+0000 7f4d6625e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60104510 0x7f4d60104930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:20.019 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.864+0000 7f4d6625e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60078e20 con 0x7f4d60104510 2024-03-11T20:46:20.019 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.864+0000 7f4d5f7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60104510 0x7f4d60104930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:20.020 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.864+0000 7f4d5f7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60104510 0x7f4d60104930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34990/0 (socket says 172.21.15.129:34990) 2024-03-11T20:46:20.020 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.864+0000 7f4d5f7fe700 1 -- 172.21.15.129:0/2119707425 learned_addr learned my addr 172.21.15.129:0/2119707425 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:20.020 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.864+0000 7f4d5f7fe700 1 -- 172.21.15.129:0/2119707425 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d60104e70 con 0x7f4d60104510 2024-03-11T20:46:20.020 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.865+0000 7f4d5f7fe700 1 --2- 172.21.15.129:0/2119707425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60104510 0x7f4d60104930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4d500098a0 tx=0x7f4d50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548ca2a067e08138 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:20.020 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.865+0000 7f4d5e7fc700 1 -- 172.21.15.129:0/2119707425 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d50004160 con 0x7f4d60104510 2024-03-11T20:46:20.020 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.865+0000 7f4d5e7fc700 1 -- 172.21.15.129:0/2119707425 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4d50004300 con 0x7f4d60104510 2024-03-11T20:46:20.020 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.865+0000 7f4d5e7fc700 1 -- 172.21.15.129:0/2119707425 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d500118f0 con 0x7f4d60104510 2024-03-11T20:46:20.020 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.865+0000 7f4d6625e700 1 -- 172.21.15.129:0/2119707425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60104510 msgr2=0x7f4d60104930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:20.020 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.865+0000 7f4d6625e700 1 --2- 172.21.15.129:0/2119707425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60104510 0x7f4d60104930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4d500098a0 tx=0x7f4d50009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:20.020 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.865+0000 7f4d6625e700 1 -- 172.21.15.129:0/2119707425 shutdown_connections 2024-03-11T20:46:20.020 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.865+0000 7f4d6625e700 1 --2- 172.21.15.129:0/2119707425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60104510 0x7f4d60104930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:20.020 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.865+0000 7f4d6625e700 1 -- 172.21.15.129:0/2119707425 >> 172.21.15.129:0/2119707425 conn(0x7f4d601000d0 msgr2=0x7f4d60102510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:20.020 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.866+0000 7f4d6625e700 1 -- 172.21.15.129:0/2119707425 shutdown_connections 2024-03-11T20:46:20.020 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.866+0000 7f4d6625e700 1 -- 172.21.15.129:0/2119707425 wait complete. 2024-03-11T20:46:20.021 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.866+0000 7f4d6625e700 1 Processor -- start 2024-03-11T20:46:20.021 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.866+0000 7f4d6625e700 1 -- start start 2024-03-11T20:46:20.021 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.866+0000 7f4d6625e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60104510 0x7f4d6019bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:20.021 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.866+0000 7f4d6625e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60105e20 con 0x7f4d60104510 2024-03-11T20:46:20.021 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.867+0000 7f4d5f7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60104510 0x7f4d6019bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:20.021 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.867+0000 7f4d5f7fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60104510 0x7f4d6019bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34996/0 (socket says 172.21.15.129:34996) 2024-03-11T20:46:20.021 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.867+0000 7f4d5f7fe700 1 -- 172.21.15.129:0/1876424851 learned_addr learned my addr 172.21.15.129:0/1876424851 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:20.021 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.867+0000 7f4d5f7fe700 1 -- 172.21.15.129:0/1876424851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6019c0c0 con 0x7f4d60104510 2024-03-11T20:46:20.021 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.867+0000 7f4d5f7fe700 1 --2- 172.21.15.129:0/1876424851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60104510 0x7f4d6019bb80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4d50011970 tx=0x7f4d50011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:20.021 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.867+0000 7f4d5cff9700 1 -- 172.21.15.129:0/1876424851 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d5001a430 con 0x7f4d60104510 2024-03-11T20:46:20.021 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.867+0000 7f4d6625e700 1 -- 172.21.15.129:0/1876424851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d6019c350 con 0x7f4d60104510 2024-03-11T20:46:20.021 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.867+0000 7f4d5cff9700 1 -- 172.21.15.129:0/1876424851 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4d5001aa50 con 0x7f4d60104510 2024-03-11T20:46:20.021 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.867+0000 7f4d5cff9700 1 -- 172.21.15.129:0/1876424851 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d50019910 con 0x7f4d60104510 2024-03-11T20:46:20.021 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.867+0000 7f4d6625e700 1 -- 172.21.15.129:0/1876424851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d6019f040 con 0x7f4d60104510 2024-03-11T20:46:20.021 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.868+0000 7f4d5cff9700 1 -- 172.21.15.129:0/1876424851 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f4d50022020 con 0x7f4d60104510 2024-03-11T20:46:20.022 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.868+0000 7f4d5cff9700 1 --2- 172.21.15.129:0/1876424851 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f4d48038ed0 0x7f4d4803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:20.022 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.868+0000 7f4d5cff9700 1 -- 172.21.15.129:0/1876424851 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4d5004df60 con 0x7f4d60104510 2024-03-11T20:46:20.022 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.868+0000 7f4d5effd700 1 --2- 172.21.15.129:0/1876424851 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f4d48038ed0 0x7f4d4803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:20.022 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.868+0000 7f4d6625e700 1 -- 172.21.15.129:0/1876424851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d60063410 con 0x7f4d60104510 2024-03-11T20:46:20.022 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.869+0000 7f4d5effd700 1 --2- 172.21.15.129:0/1876424851 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f4d48038ed0 0x7f4d4803b390 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4d54006fd0 tx=0x7f4d54006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:20.022 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:18.873+0000 7f4d5cff9700 1 -- 172.21.15.129:0/1876424851 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f4d5004dcc0 con 0x7f4d60104510 2024-03-11T20:46:20.022 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:19.088+0000 7f4d6625e700 1 -- 172.21.15.129:0/1876424851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f4d60194ec0 con 0x7f4d60104510 2024-03-11T20:46:20.022 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:19.093+0000 7f4d5cff9700 1 -- 172.21.15.129:0/1876424851 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f4d5002c8c0 con 0x7f4d60104510 2024-03-11T20:46:20.022 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:19.095+0000 7f4d6625e700 1 -- 172.21.15.129:0/1876424851 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f4d48038ed0 msgr2=0x7f4d4803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:20.022 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:19.095+0000 7f4d6625e700 1 --2- 172.21.15.129:0/1876424851 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f4d48038ed0 0x7f4d4803b390 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4d54006fd0 tx=0x7f4d54006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:20.022 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:19.095+0000 7f4d6625e700 1 -- 172.21.15.129:0/1876424851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60104510 msgr2=0x7f4d6019bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:20.022 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:19.095+0000 7f4d6625e700 1 --2- 172.21.15.129:0/1876424851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60104510 0x7f4d6019bb80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4d50011970 tx=0x7f4d50011a50 comp rx=0 tx=0).stop 2024-03-11T20:46:20.023 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:19.095+0000 7f4d6625e700 1 -- 172.21.15.129:0/1876424851 shutdown_connections 2024-03-11T20:46:20.023 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:19.095+0000 7f4d6625e700 1 --2- 172.21.15.129:0/1876424851 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f4d48038ed0 0x7f4d4803b390 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:20.023 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:19.095+0000 7f4d6625e700 1 --2- 172.21.15.129:0/1876424851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60104510 0x7f4d6019bb80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:20.023 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:19.095+0000 7f4d6625e700 1 -- 172.21.15.129:0/1876424851 >> 172.21.15.129:0/1876424851 conn(0x7f4d601000d0 msgr2=0x7f4d6018fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:20.023 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:19.096+0000 7f4d6625e700 1 -- 172.21.15.129:0/1876424851 shutdown_connections 2024-03-11T20:46:20.023 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:19.096+0000 7f4d6625e700 1 -- 172.21.15.129:0/1876424851 wait complete. 2024-03-11T20:46:20.323 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1876424851' entity='client.admin' 2024-03-11T20:46:21.259 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.459+0000 7fac3b8f6700 1 Processor -- start 2024-03-11T20:46:21.259 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.459+0000 7fac3b8f6700 1 -- start start 2024-03-11T20:46:21.259 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.460+0000 7fac3b8f6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac34106780 0x7fac34108b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:21.259 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.460+0000 7fac3b8f6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac34078dc0 con 0x7fac34106780 2024-03-11T20:46:21.259 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.460+0000 7fac39692700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac34106780 0x7fac34108b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:21.259 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.460+0000 7fac39692700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac34106780 0x7fac34108b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34998/0 (socket says 172.21.15.129:34998) 2024-03-11T20:46:21.259 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.460+0000 7fac39692700 1 -- 172.21.15.129:0/2650898844 learned_addr learned my addr 172.21.15.129:0/2650898844 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:21.259 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.460+0000 7fac39692700 1 -- 172.21.15.129:0/2650898844 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac341090b0 con 0x7fac34106780 2024-03-11T20:46:21.260 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.460+0000 7fac39692700 1 --2- 172.21.15.129:0/2650898844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac34106780 0x7fac34108b70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fac240098a0 tx=0x7fac24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207be690cad0916 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:21.260 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.460+0000 7fac2bfff700 1 -- 172.21.15.129:0/2650898844 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac24004160 con 0x7fac34106780 2024-03-11T20:46:21.260 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.460+0000 7fac2bfff700 1 -- 172.21.15.129:0/2650898844 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fac24004300 con 0x7fac34106780 2024-03-11T20:46:21.260 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.460+0000 7fac2bfff700 1 -- 172.21.15.129:0/2650898844 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac240118f0 con 0x7fac34106780 2024-03-11T20:46:21.260 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.461+0000 7fac3b8f6700 1 -- 172.21.15.129:0/2650898844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac34106780 msgr2=0x7fac34108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:21.260 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.461+0000 7fac3b8f6700 1 --2- 172.21.15.129:0/2650898844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac34106780 0x7fac34108b70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fac240098a0 tx=0x7fac24009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:21.260 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.461+0000 7fac3b8f6700 1 -- 172.21.15.129:0/2650898844 shutdown_connections 2024-03-11T20:46:21.260 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.461+0000 7fac3b8f6700 1 --2- 172.21.15.129:0/2650898844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac34106780 0x7fac34108b70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:21.260 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.461+0000 7fac3b8f6700 1 -- 172.21.15.129:0/2650898844 >> 172.21.15.129:0/2650898844 conn(0x7fac341000d0 msgr2=0x7fac34102510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:21.260 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.461+0000 7fac3b8f6700 1 -- 172.21.15.129:0/2650898844 shutdown_connections 2024-03-11T20:46:21.260 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.461+0000 7fac3b8f6700 1 -- 172.21.15.129:0/2650898844 wait complete. 2024-03-11T20:46:21.260 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.462+0000 7fac3b8f6700 1 Processor -- start 2024-03-11T20:46:21.260 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.462+0000 7fac3b8f6700 1 -- start start 2024-03-11T20:46:21.260 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.462+0000 7fac3b8f6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac34106780 0x7fac34106000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:21.261 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.462+0000 7fac3b8f6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac34109f30 con 0x7fac34106780 2024-03-11T20:46:21.261 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.462+0000 7fac39692700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac34106780 0x7fac34106000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:21.261 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.462+0000 7fac39692700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac34106780 0x7fac34106000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35014/0 (socket says 172.21.15.129:35014) 2024-03-11T20:46:21.261 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.462+0000 7fac39692700 1 -- 172.21.15.129:0/3898547365 learned_addr learned my addr 172.21.15.129:0/3898547365 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:21.261 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.462+0000 7fac39692700 1 -- 172.21.15.129:0/3898547365 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac34106540 con 0x7fac34106780 2024-03-11T20:46:21.261 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.462+0000 7fac39692700 1 --2- 172.21.15.129:0/3898547365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac34106780 0x7fac34106000 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fac24011970 tx=0x7fac24011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:21.261 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.463+0000 7fac2a7fc700 1 -- 172.21.15.129:0/3898547365 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac2401a430 con 0x7fac34106780 2024-03-11T20:46:21.261 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.463+0000 7fac2a7fc700 1 -- 172.21.15.129:0/3898547365 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fac2401aa50 con 0x7fac34106780 2024-03-11T20:46:21.261 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.463+0000 7fac3b8f6700 1 -- 172.21.15.129:0/3898547365 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac34104730 con 0x7fac34106780 2024-03-11T20:46:21.261 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.463+0000 7fac2a7fc700 1 -- 172.21.15.129:0/3898547365 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac24019910 con 0x7fac34106780 2024-03-11T20:46:21.261 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.463+0000 7fac3b8f6700 1 -- 172.21.15.129:0/3898547365 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac34104c70 con 0x7fac34106780 2024-03-11T20:46:21.261 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.464+0000 7fac2a7fc700 1 -- 172.21.15.129:0/3898547365 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fac24022020 con 0x7fac34106780 2024-03-11T20:46:21.262 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.464+0000 7fac3b8f6700 1 -- 172.21.15.129:0/3898547365 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac34063410 con 0x7fac34106780 2024-03-11T20:46:21.262 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.464+0000 7fac2a7fc700 1 --2- 172.21.15.129:0/3898547365 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fac20038f20 0x7fac2003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:21.262 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.464+0000 7fac2a7fc700 1 -- 172.21.15.129:0/3898547365 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fac2404d150 con 0x7fac34106780 2024-03-11T20:46:21.262 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.464+0000 7fac38e91700 1 --2- 172.21.15.129:0/3898547365 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fac20038f20 0x7fac2003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:21.262 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.464+0000 7fac38e91700 1 --2- 172.21.15.129:0/3898547365 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fac20038f20 0x7fac2003b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fac30006fd0 tx=0x7fac30006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:21.262 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.468+0000 7fac2a7fc700 1 -- 172.21.15.129:0/3898547365 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fac2402b350 con 0x7fac34106780 2024-03-11T20:46:21.262 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.662+0000 7fac3b8f6700 1 -- 172.21.15.129:0/3898547365 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fac341053d0 con 0x7fac34106780 2024-03-11T20:46:21.262 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.664+0000 7fac2a7fc700 1 -- 172.21.15.129:0/3898547365 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fac2404ceb0 con 0x7fac34106780 2024-03-11T20:46:21.262 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.667+0000 7fac3b8f6700 1 -- 172.21.15.129:0/3898547365 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fac20038f20 msgr2=0x7fac2003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:21.262 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.667+0000 7fac3b8f6700 1 --2- 172.21.15.129:0/3898547365 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fac20038f20 0x7fac2003b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fac30006fd0 tx=0x7fac30006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:21.262 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.667+0000 7fac3b8f6700 1 -- 172.21.15.129:0/3898547365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac34106780 msgr2=0x7fac34106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:21.262 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.667+0000 7fac3b8f6700 1 --2- 172.21.15.129:0/3898547365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac34106780 0x7fac34106000 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fac24011970 tx=0x7fac24011a50 comp rx=0 tx=0).stop 2024-03-11T20:46:21.262 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.667+0000 7fac3b8f6700 1 -- 172.21.15.129:0/3898547365 shutdown_connections 2024-03-11T20:46:21.263 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.667+0000 7fac3b8f6700 1 --2- 172.21.15.129:0/3898547365 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7fac20038f20 0x7fac2003b3e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:21.263 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.667+0000 7fac3b8f6700 1 --2- 172.21.15.129:0/3898547365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac34106780 0x7fac34106000 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:21.263 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.667+0000 7fac3b8f6700 1 -- 172.21.15.129:0/3898547365 >> 172.21.15.129:0/3898547365 conn(0x7fac341000d0 msgr2=0x7fac34100c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:21.263 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.667+0000 7fac3b8f6700 1 -- 172.21.15.129:0/3898547365 shutdown_connections 2024-03-11T20:46:21.263 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:20.667+0000 7fac3b8f6700 1 -- 172.21.15.129:0/3898547365 wait complete. 2024-03-11T20:46:21.263 INFO:teuthology.orchestra.run.smithi129.stdout:Enabling the dashboard module... 2024-03-11T20:46:21.779 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3898547365' entity='client.admin' 2024-03-11T20:46:21.779 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:21 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' 2024-03-11T20:46:22.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4087742976' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T20:46:22.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:22 smithi129 ceph-mon[30184]: from='mgr.14120 172.21.15.129:0/1799172450' entity='mgr.y' 2024-03-11T20:46:22.978 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:22 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ignoring --setuser ceph since I am not root 2024-03-11T20:46:22.978 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:22 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ignoring --setgroup ceph since I am not root 2024-03-11T20:46:23.375 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:23 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:23.251+0000 7fe7efa56200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T20:46:23.375 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:23 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:23.345+0000 7fe7efa56200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T20:46:23.377 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.740+0000 7f3f3a5d4700 1 Processor -- start 2024-03-11T20:46:23.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.740+0000 7f3f3a5d4700 1 -- start start 2024-03-11T20:46:23.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.740+0000 7f3f3a5d4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f34104b60 0x7f3f34104f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:23.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.741+0000 7f3f3a5d4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f34105550 con 0x7f3f34104b60 2024-03-11T20:46:23.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.741+0000 7f3f33fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f34104b60 0x7f3f34104f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:23.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.741+0000 7f3f33fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f34104b60 0x7f3f34104f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35020/0 (socket says 172.21.15.129:35020) 2024-03-11T20:46:23.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.741+0000 7f3f33fff700 1 -- 172.21.15.129:0/348054386 learned_addr learned my addr 172.21.15.129:0/348054386 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:23.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.741+0000 7f3f33fff700 1 -- 172.21.15.129:0/348054386 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f34105d60 con 0x7f3f34104b60 2024-03-11T20:46:23.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.741+0000 7f3f33fff700 1 --2- 172.21.15.129:0/348054386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f34104b60 0x7f3f34104f80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f240098a0 tx=0x7f3f24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f4da974ca7f1f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:23.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.741+0000 7f3f32ffd700 1 -- 172.21.15.129:0/348054386 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f24004160 con 0x7f3f34104b60 2024-03-11T20:46:23.383 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.741+0000 7f3f32ffd700 1 -- 172.21.15.129:0/348054386 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3f24004300 con 0x7f3f34104b60 2024-03-11T20:46:23.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.741+0000 7f3f32ffd700 1 -- 172.21.15.129:0/348054386 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f240118f0 con 0x7f3f34104b60 2024-03-11T20:46:23.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.742+0000 7f3f3a5d4700 1 -- 172.21.15.129:0/348054386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f34104b60 msgr2=0x7f3f34104f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:23.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.742+0000 7f3f3a5d4700 1 --2- 172.21.15.129:0/348054386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f34104b60 0x7f3f34104f80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3f240098a0 tx=0x7f3f24009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:23.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.742+0000 7f3f3a5d4700 1 -- 172.21.15.129:0/348054386 shutdown_connections 2024-03-11T20:46:23.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.742+0000 7f3f3a5d4700 1 --2- 172.21.15.129:0/348054386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f34104b60 0x7f3f34104f80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:23.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.742+0000 7f3f3a5d4700 1 -- 172.21.15.129:0/348054386 >> 172.21.15.129:0/348054386 conn(0x7f3f341000e0 msgr2=0x7f3f34102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:23.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.742+0000 7f3f3a5d4700 1 -- 172.21.15.129:0/348054386 shutdown_connections 2024-03-11T20:46:23.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.742+0000 7f3f3a5d4700 1 -- 172.21.15.129:0/348054386 wait complete. 2024-03-11T20:46:23.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.743+0000 7f3f3a5d4700 1 Processor -- start 2024-03-11T20:46:23.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.743+0000 7f3f3a5d4700 1 -- start start 2024-03-11T20:46:23.384 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.744+0000 7f3f3a5d4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f34104b60 0x7f3f34198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:23.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.744+0000 7f3f3a5d4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f34108f90 con 0x7f3f34104b60 2024-03-11T20:46:23.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.744+0000 7f3f33fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f34104b60 0x7f3f34198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:23.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.744+0000 7f3f33fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f34104b60 0x7f3f34198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35032/0 (socket says 172.21.15.129:35032) 2024-03-11T20:46:23.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.744+0000 7f3f33fff700 1 -- 172.21.15.129:0/4087742976 learned_addr learned my addr 172.21.15.129:0/4087742976 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:23.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.744+0000 7f3f33fff700 1 -- 172.21.15.129:0/4087742976 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f34198e70 con 0x7f3f34104b60 2024-03-11T20:46:23.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.744+0000 7f3f33fff700 1 --2- 172.21.15.129:0/4087742976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f34104b60 0x7f3f34198930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009fb0 tx=0x7f3f24011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:23.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.744+0000 7f3f317fa700 1 -- 172.21.15.129:0/4087742976 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f2401a430 con 0x7f3f34104b60 2024-03-11T20:46:23.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.744+0000 7f3f317fa700 1 -- 172.21.15.129:0/4087742976 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3f2401aa50 con 0x7f3f34104b60 2024-03-11T20:46:23.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.744+0000 7f3f317fa700 1 -- 172.21.15.129:0/4087742976 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f24019800 con 0x7f3f34104b60 2024-03-11T20:46:23.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.744+0000 7f3f3a5d4700 1 -- 172.21.15.129:0/4087742976 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f34199100 con 0x7f3f34104b60 2024-03-11T20:46:23.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.744+0000 7f3f3a5d4700 1 -- 172.21.15.129:0/4087742976 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3419bdf0 con 0x7f3f34104b60 2024-03-11T20:46:23.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.745+0000 7f3f317fa700 1 -- 172.21.15.129:0/4087742976 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f3f24022020 con 0x7f3f34104b60 2024-03-11T20:46:23.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.745+0000 7f3f3a5d4700 1 -- 172.21.15.129:0/4087742976 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f34063410 con 0x7f3f34104b60 2024-03-11T20:46:23.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.746+0000 7f3f317fa700 1 --2- 172.21.15.129:0/4087742976 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f3f1c038f20 0x7f3f1c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:23.385 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.746+0000 7f3f317fa700 1 -- 172.21.15.129:0/4087742976 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3f2404d120 con 0x7f3f34104b60 2024-03-11T20:46:23.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.746+0000 7f3f337fe700 1 --2- 172.21.15.129:0/4087742976 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f3f1c038f20 0x7f3f1c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:23.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.746+0000 7f3f337fe700 1 --2- 172.21.15.129:0/4087742976 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f3f1c038f20 0x7f3f1c03b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3f28006fd0 tx=0x7f3f28006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:23.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:21.750+0000 7f3f317fa700 1 -- 172.21.15.129:0/4087742976 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f3f2404ce80 con 0x7f3f34104b60 2024-03-11T20:46:23.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:22.016+0000 7f3f3a5d4700 1 -- 172.21.15.129:0/4087742976 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f3f3419c1d0 con 0x7f3f34104b60 2024-03-11T20:46:23.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:22.669+0000 7f3f317fa700 1 -- 172.21.15.129:0/4087742976 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f3f2402b990 con 0x7f3f34104b60 2024-03-11T20:46:23.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:22.670+0000 7f3f317fa700 1 -- 172.21.15.129:0/4087742976 <== mon.0 v2:172.21.15.129:3300/0 8 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f3f2402c720 con 0x7f3f34104b60 2024-03-11T20:46:23.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:22.672+0000 7f3f3a5d4700 1 -- 172.21.15.129:0/4087742976 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f3f1c038f20 msgr2=0x7f3f1c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:23.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:22.672+0000 7f3f3a5d4700 1 --2- 172.21.15.129:0/4087742976 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f3f1c038f20 0x7f3f1c03b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3f28006fd0 tx=0x7f3f28006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:23.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:22.672+0000 7f3f3a5d4700 1 -- 172.21.15.129:0/4087742976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f34104b60 msgr2=0x7f3f34198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:23.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:22.672+0000 7f3f3a5d4700 1 --2- 172.21.15.129:0/4087742976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f34104b60 0x7f3f34198930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009fb0 tx=0x7f3f24011a20 comp rx=0 tx=0).stop 2024-03-11T20:46:23.386 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:22.672+0000 7f3f3a5d4700 1 -- 172.21.15.129:0/4087742976 shutdown_connections 2024-03-11T20:46:23.387 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:22.672+0000 7f3f3a5d4700 1 --2- 172.21.15.129:0/4087742976 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f3f1c038f20 0x7f3f1c03b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:23.387 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:22.672+0000 7f3f3a5d4700 1 --2- 172.21.15.129:0/4087742976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f34104b60 0x7f3f34198930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:23.387 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:22.672+0000 7f3f3a5d4700 1 -- 172.21.15.129:0/4087742976 >> 172.21.15.129:0/4087742976 conn(0x7f3f341000e0 msgr2=0x7f3f34100c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:23.387 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:22.672+0000 7f3f3a5d4700 1 -- 172.21.15.129:0/4087742976 shutdown_connections 2024-03-11T20:46:23.387 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:22.672+0000 7f3f3a5d4700 1 -- 172.21.15.129:0/4087742976 wait complete. 2024-03-11T20:46:23.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4087742976' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T20:46:23.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:23 smithi129 ceph-mon[30184]: mgrmap e9: y(active, since 20s) 2024-03-11T20:46:24.256 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:24 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:24.015+0000 7fe7efa56200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T20:46:24.256 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:24 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:24.110+0000 7fe7efa56200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T20:46:24.664 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:24 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:24.262+0000 7fe7efa56200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T20:46:24.665 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:24 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:24.370+0000 7fe7efa56200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T20:46:24.665 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:24 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:24.453+0000 7fe7efa56200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T20:46:24.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1227730499' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T20:46:24.977 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:24 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:24.722+0000 7fe7efa56200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T20:46:24.977 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:24 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:24.833+0000 7fe7efa56200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T20:46:24.977 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:24 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:24.918+0000 7fe7efa56200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T20:46:25.027 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout { 2024-03-11T20:46:25.027 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-11T20:46:25.027 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T20:46:25.027 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T20:46:25.027 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T20:46:25.027 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout } 2024-03-11T20:46:25.027 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.005+0000 7ff896c0d700 1 Processor -- start 2024-03-11T20:46:25.027 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.005+0000 7ff896c0d700 1 -- start start 2024-03-11T20:46:25.027 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.005+0000 7ff896c0d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff890105030 0x7ff890105450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:25.027 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.005+0000 7ff896c0d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff890105a20 con 0x7ff890105030 2024-03-11T20:46:25.027 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.006+0000 7ff8949a9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff890105030 0x7ff890105450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:25.028 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.006+0000 7ff8949a9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff890105030 0x7ff890105450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35058/0 (socket says 172.21.15.129:35058) 2024-03-11T20:46:25.028 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.006+0000 7ff8949a9700 1 -- 172.21.15.129:0/337668301 learned_addr learned my addr 172.21.15.129:0/337668301 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:25.028 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.006+0000 7ff8949a9700 1 -- 172.21.15.129:0/337668301 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff890105d50 con 0x7ff890105030 2024-03-11T20:46:25.028 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.006+0000 7ff8949a9700 1 --2- 172.21.15.129:0/337668301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff890105030 0x7ff890105450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff8840098a0 tx=0x7ff884009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=285fa17d9b48937d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:25.028 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.006+0000 7ff88f7fe700 1 -- 172.21.15.129:0/337668301 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff884004160 con 0x7ff890105030 2024-03-11T20:46:25.028 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.006+0000 7ff88f7fe700 1 -- 172.21.15.129:0/337668301 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ff884004300 con 0x7ff890105030 2024-03-11T20:46:25.028 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.006+0000 7ff88f7fe700 1 -- 172.21.15.129:0/337668301 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff8840118f0 con 0x7ff890105030 2024-03-11T20:46:25.028 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.007+0000 7ff896c0d700 1 -- 172.21.15.129:0/337668301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff890105030 msgr2=0x7ff890105450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:25.028 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.007+0000 7ff896c0d700 1 --2- 172.21.15.129:0/337668301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff890105030 0x7ff890105450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff8840098a0 tx=0x7ff884009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:25.028 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.007+0000 7ff896c0d700 1 -- 172.21.15.129:0/337668301 shutdown_connections 2024-03-11T20:46:25.028 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.007+0000 7ff896c0d700 1 --2- 172.21.15.129:0/337668301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff890105030 0x7ff890105450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:25.028 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.007+0000 7ff896c0d700 1 -- 172.21.15.129:0/337668301 >> 172.21.15.129:0/337668301 conn(0x7ff8900ff850 msgr2=0x7ff890101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:25.028 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.007+0000 7ff896c0d700 1 -- 172.21.15.129:0/337668301 shutdown_connections 2024-03-11T20:46:25.029 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.007+0000 7ff896c0d700 1 -- 172.21.15.129:0/337668301 wait complete. 2024-03-11T20:46:25.029 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.007+0000 7ff896c0d700 1 Processor -- start 2024-03-11T20:46:25.029 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.008+0000 7ff896c0d700 1 -- start start 2024-03-11T20:46:25.029 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.008+0000 7ff896c0d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff890105030 0x7ff890199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:25.029 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.008+0000 7ff896c0d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff890108f80 con 0x7ff890105030 2024-03-11T20:46:25.029 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.008+0000 7ff8949a9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff890105030 0x7ff890199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:25.029 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.008+0000 7ff8949a9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff890105030 0x7ff890199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35074/0 (socket says 172.21.15.129:35074) 2024-03-11T20:46:25.029 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.008+0000 7ff8949a9700 1 -- 172.21.15.129:0/1227730499 learned_addr learned my addr 172.21.15.129:0/1227730499 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:25.029 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.008+0000 7ff8949a9700 1 -- 172.21.15.129:0/1227730499 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff890199f40 con 0x7ff890105030 2024-03-11T20:46:25.029 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.008+0000 7ff8949a9700 1 --2- 172.21.15.129:0/1227730499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff890105030 0x7ff890199a00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff884009870 tx=0x7ff884011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:25.029 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.008+0000 7ff88dffb700 1 -- 172.21.15.129:0/1227730499 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff8840038b0 con 0x7ff890105030 2024-03-11T20:46:25.029 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.008+0000 7ff88dffb700 1 -- 172.21.15.129:0/1227730499 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ff884003a50 con 0x7ff890105030 2024-03-11T20:46:25.029 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.008+0000 7ff88dffb700 1 -- 172.21.15.129:0/1227730499 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff884024410 con 0x7ff890105030 2024-03-11T20:46:25.030 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.008+0000 7ff896c0d700 1 -- 172.21.15.129:0/1227730499 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89019a1d0 con 0x7ff890105030 2024-03-11T20:46:25.030 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.009+0000 7ff896c0d700 1 -- 172.21.15.129:0/1227730499 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff89019cec0 con 0x7ff890105030 2024-03-11T20:46:25.030 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.009+0000 7ff88dffb700 1 -- 172.21.15.129:0/1227730499 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7ff884022020 con 0x7ff890105030 2024-03-11T20:46:25.030 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.010+0000 7ff896c0d700 1 -- 172.21.15.129:0/1227730499 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff890063410 con 0x7ff890105030 2024-03-11T20:46:25.030 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.010+0000 7ff88dffb700 1 --2- 172.21.15.129:0/1227730499 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7ff880038fd0 0x7ff88003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:25.030 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.010+0000 7ff88dffb700 1 -- 172.21.15.129:0/1227730499 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff88404cfc0 con 0x7ff890105030 2024-03-11T20:46:25.030 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.010+0000 7ff88ffff700 1 -- 172.21.15.129:0/1227730499 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7ff880038fd0 msgr2=0x7ff88003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/521270824 2024-03-11T20:46:25.030 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.010+0000 7ff88ffff700 1 --2- 172.21.15.129:0/1227730499 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7ff880038fd0 0x7ff88003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T20:46:25.030 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.014+0000 7ff88dffb700 1 -- 172.21.15.129:0/1227730499 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7ff884017ca0 con 0x7ff890105030 2024-03-11T20:46:25.030 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.211+0000 7ff88ffff700 1 -- 172.21.15.129:0/1227730499 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7ff880038fd0 msgr2=0x7ff88003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/521270824 2024-03-11T20:46:25.030 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.211+0000 7ff88ffff700 1 --2- 172.21.15.129:0/1227730499 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7ff880038fd0 0x7ff88003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T20:46:25.030 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.275+0000 7ff896c0d700 1 -- 172.21.15.129:0/1227730499 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7ff89002d8d0 con 0x7ff890105030 2024-03-11T20:46:25.031 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.275+0000 7ff88dffb700 1 -- 172.21.15.129:0/1227730499 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7ff88402d3f0 con 0x7ff890105030 2024-03-11T20:46:25.031 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.277+0000 7ff896c0d700 1 -- 172.21.15.129:0/1227730499 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7ff880038fd0 msgr2=0x7ff88003b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T20:46:25.031 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.277+0000 7ff896c0d700 1 --2- 172.21.15.129:0/1227730499 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7ff880038fd0 0x7ff88003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:25.031 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.277+0000 7ff896c0d700 1 -- 172.21.15.129:0/1227730499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff890105030 msgr2=0x7ff890199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:25.031 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.277+0000 7ff896c0d700 1 --2- 172.21.15.129:0/1227730499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff890105030 0x7ff890199a00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff884009870 tx=0x7ff884011ef0 comp rx=0 tx=0).stop 2024-03-11T20:46:25.031 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.277+0000 7ff896c0d700 1 -- 172.21.15.129:0/1227730499 shutdown_connections 2024-03-11T20:46:25.031 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.277+0000 7ff896c0d700 1 --2- 172.21.15.129:0/1227730499 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7ff880038fd0 0x7ff88003b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:25.031 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.277+0000 7ff896c0d700 1 --2- 172.21.15.129:0/1227730499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff890105030 0x7ff890199a00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:25.031 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.277+0000 7ff896c0d700 1 -- 172.21.15.129:0/1227730499 >> 172.21.15.129:0/1227730499 conn(0x7ff8900ff850 msgr2=0x7ff8901006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:25.031 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.277+0000 7ff896c0d700 1 -- 172.21.15.129:0/1227730499 shutdown_connections 2024-03-11T20:46:25.031 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:24.277+0000 7ff896c0d700 1 -- 172.21.15.129:0/1227730499 wait complete. 2024-03-11T20:46:25.031 INFO:teuthology.orchestra.run.smithi129.stdout:Waiting for the mgr to restart... 2024-03-11T20:46:25.031 INFO:teuthology.orchestra.run.smithi129.stdout:Waiting for mgr epoch 9... 2024-03-11T20:46:25.268 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:25 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:25.013+0000 7fe7efa56200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T20:46:25.268 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:25 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:25.197+0000 7fe7efa56200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T20:46:26.477 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:26.175+0000 7fe7efa56200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T20:46:26.477 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:26.304+0000 7fe7efa56200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T20:46:26.916 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:26.637+0000 7fe7efa56200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T20:46:26.917 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:26.727+0000 7fe7efa56200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T20:46:26.917 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:26.921+0000 7fe7efa56200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:46:27.227 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:27 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:27.107+0000 7fe7efa56200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T20:46:27.645 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:27 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:27.390+0000 7fe7efa56200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:46:27.645 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:27 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:27.650+0000 7fe7efa56200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T20:46:27.977 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:27 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:27.884+0000 7fe7efa56200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T20:46:28.977 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:28 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:28.675+0000 7fe7efa56200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T20:46:29.554 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:29 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:29.245+0000 7fe7efa56200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T20:46:29.554 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:29 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:29.343+0000 7fe7efa56200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T20:46:29.555 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:46:29 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:46:29.559+0000 7fe7efa56200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T20:46:29.805 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:29 smithi129 ceph-mon[30184]: Active manager daemon y restarted 2024-03-11T20:46:29.805 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:29 smithi129 ceph-mon[30184]: Activating manager daemon y 2024-03-11T20:46:29.805 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:29 smithi129 ceph-mon[30184]: osdmap e3: 0 total, 0 up, 0 in 2024-03-11T20:46:29.805 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:29 smithi129 ceph-mon[30184]: mgrmap e10: y(active, starting, since 0.00319763s) 2024-03-11T20:46:29.805 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:46:29.805 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T20:46:29.805 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T20:46:29.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T20:46:29.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T20:46:29.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:29 smithi129 ceph-mon[30184]: Manager daemon y is now available 2024-03-11T20:46:29.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:29.806 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:30.670 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:30 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T20:46:30.670 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:30 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T20:46:30.670 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:30 smithi129 ceph-mon[30184]: mgrmap e11: y(active, since 1.00651s) 2024-03-11T20:46:31.516 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout { 2024-03-11T20:46:31.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-11T20:46:31.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T20:46:31.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout } 2024-03-11T20:46:31.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.521+0000 7f6f041c0700 1 Processor -- start 2024-03-11T20:46:31.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.521+0000 7f6f041c0700 1 -- start start 2024-03-11T20:46:31.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.522+0000 7f6f041c0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6efc07b720 0x7f6efc079b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:31.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.522+0000 7f6f041c0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc07a150 con 0x7f6efc07b720 2024-03-11T20:46:31.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.522+0000 7f6f01f5c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6efc07b720 0x7f6efc079b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:31.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.522+0000 7f6f01f5c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6efc07b720 0x7f6efc079b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57578/0 (socket says 172.21.15.129:57578) 2024-03-11T20:46:31.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.522+0000 7f6f01f5c700 1 -- 172.21.15.129:0/1389516392 learned_addr learned my addr 172.21.15.129:0/1389516392 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:31.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.522+0000 7f6f01f5c700 1 -- 172.21.15.129:0/1389516392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc07a2d0 con 0x7f6efc07b720 2024-03-11T20:46:31.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.522+0000 7f6f01f5c700 1 --2- 172.21.15.129:0/1389516392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6efc07b720 0x7f6efc079b80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6eec0098a0 tx=0x7f6eec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d5e79d6f5bdb49 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:31.517 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.522+0000 7f6f00f5a700 1 -- 172.21.15.129:0/1389516392 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6eec004160 con 0x7f6efc07b720 2024-03-11T20:46:31.518 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.522+0000 7f6f00f5a700 1 -- 172.21.15.129:0/1389516392 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6eec004300 con 0x7f6efc07b720 2024-03-11T20:46:31.518 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.522+0000 7f6f00f5a700 1 -- 172.21.15.129:0/1389516392 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6eec0118f0 con 0x7f6efc07b720 2024-03-11T20:46:31.518 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.523+0000 7f6f041c0700 1 -- 172.21.15.129:0/1389516392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6efc07b720 msgr2=0x7f6efc079b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:31.518 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.523+0000 7f6f041c0700 1 --2- 172.21.15.129:0/1389516392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6efc07b720 0x7f6efc079b80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6eec0098a0 tx=0x7f6eec009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:31.518 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.523+0000 7f6f041c0700 1 -- 172.21.15.129:0/1389516392 shutdown_connections 2024-03-11T20:46:31.518 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.523+0000 7f6f041c0700 1 --2- 172.21.15.129:0/1389516392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6efc07b720 0x7f6efc079b80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:31.518 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.523+0000 7f6f041c0700 1 -- 172.21.15.129:0/1389516392 >> 172.21.15.129:0/1389516392 conn(0x7f6efc1011c0 msgr2=0x7f6efc103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:31.518 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.523+0000 7f6f041c0700 1 -- 172.21.15.129:0/1389516392 shutdown_connections 2024-03-11T20:46:31.518 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.523+0000 7f6f041c0700 1 -- 172.21.15.129:0/1389516392 wait complete. 2024-03-11T20:46:31.518 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.524+0000 7f6f041c0700 1 Processor -- start 2024-03-11T20:46:31.518 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.524+0000 7f6f041c0700 1 -- start start 2024-03-11T20:46:31.518 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.524+0000 7f6f041c0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6efc07b720 0x7f6efc19bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:31.518 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.524+0000 7f6f041c0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc10c1a0 con 0x7f6efc07b720 2024-03-11T20:46:31.519 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.524+0000 7f6f01f5c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6efc07b720 0x7f6efc19bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:31.519 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.524+0000 7f6f01f5c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6efc07b720 0x7f6efc19bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57590/0 (socket says 172.21.15.129:57590) 2024-03-11T20:46:31.519 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.524+0000 7f6f01f5c700 1 -- 172.21.15.129:0/3304565361 learned_addr learned my addr 172.21.15.129:0/3304565361 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:31.519 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.525+0000 7f6f01f5c700 1 -- 172.21.15.129:0/3304565361 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc19c0e0 con 0x7f6efc07b720 2024-03-11T20:46:31.519 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.525+0000 7f6f01f5c700 1 --2- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6efc07b720 0x7f6efc19bba0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6eec003f60 tx=0x7f6eec0119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:31.519 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.525+0000 7f6ef2ffd700 1 -- 172.21.15.129:0/3304565361 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6eec003a40 con 0x7f6efc07b720 2024-03-11T20:46:31.519 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.525+0000 7f6f041c0700 1 -- 172.21.15.129:0/3304565361 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6efc19c370 con 0x7f6efc07b720 2024-03-11T20:46:31.519 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.525+0000 7f6ef2ffd700 1 -- 172.21.15.129:0/3304565361 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6eec0248b0 con 0x7f6efc07b720 2024-03-11T20:46:31.519 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.525+0000 7f6ef2ffd700 1 -- 172.21.15.129:0/3304565361 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6eec01a780 con 0x7f6efc07b720 2024-03-11T20:46:31.519 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.525+0000 7f6f041c0700 1 -- 172.21.15.129:0/3304565361 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6efc19f060 con 0x7f6efc07b720 2024-03-11T20:46:31.519 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.526+0000 7f6ef2ffd700 1 -- 172.21.15.129:0/3304565361 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f6eec01aa80 con 0x7f6efc07b720 2024-03-11T20:46:31.519 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.526+0000 7f6ef2ffd700 1 --2- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f6ee8038f20 0x7f6ee803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:31.519 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.526+0000 7f6ef2ffd700 1 -- 172.21.15.129:0/3304565361 --> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ee803baf0 con 0x7f6ee8038f20 2024-03-11T20:46:31.519 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.526+0000 7f6ef2ffd700 1 -- 172.21.15.129:0/3304565361 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6eec04d3a0 con 0x7f6efc07b720 2024-03-11T20:46:31.520 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.526+0000 7f6f0175b700 1 -- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f6ee8038f20 msgr2=0x7f6ee803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/521270824 2024-03-11T20:46:31.520 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.526+0000 7f6f0175b700 1 --2- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f6ee8038f20 0x7f6ee803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T20:46:31.520 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.727+0000 7f6f0175b700 1 -- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f6ee8038f20 msgr2=0x7f6ee803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/521270824 2024-03-11T20:46:31.520 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:25.727+0000 7f6f0175b700 1 --2- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f6ee8038f20 0x7f6ee803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T20:46:31.520 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:26.128+0000 7f6f0175b700 1 -- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f6ee8038f20 msgr2=0x7f6ee803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/521270824 2024-03-11T20:46:31.520 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:26.128+0000 7f6f0175b700 1 --2- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f6ee8038f20 0x7f6ee803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-11T20:46:31.520 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:26.929+0000 7f6f0175b700 1 -- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f6ee8038f20 msgr2=0x7f6ee803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/521270824 2024-03-11T20:46:31.520 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:26.929+0000 7f6f0175b700 1 --2- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f6ee8038f20 0x7f6ee803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-11T20:46:31.520 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:28.530+0000 7f6f0175b700 1 -- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f6ee8038f20 msgr2=0x7f6ee803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/521270824 2024-03-11T20:46:31.520 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:28.530+0000 7f6f0175b700 1 --2- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f6ee8038f20 0x7f6ee803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-11T20:46:31.520 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:29.575+0000 7f6ef2ffd700 1 -- 172.21.15.129:0/3304565361 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7f6eec02d910 con 0x7f6efc07b720 2024-03-11T20:46:31.520 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:29.575+0000 7f6ef2ffd700 1 -- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f6ee8038f20 msgr2=0x7f6ee803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T20:46:31.520 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:29.575+0000 7f6ef2ffd700 1 --2- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f6ee8038f20 0x7f6ee803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:31.521 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.577+0000 7f6ef2ffd700 1 -- 172.21.15.129:0/3304565361 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f6eec04f400 con 0x7f6efc07b720 2024-03-11T20:46:31.521 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.577+0000 7f6ef2ffd700 1 --2- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6ee803c9a0 0x7f6ee803ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:31.521 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.577+0000 7f6ef2ffd700 1 -- 172.21.15.129:0/3304565361 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ee803baf0 con 0x7f6ee803c9a0 2024-03-11T20:46:31.521 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.577+0000 7f6f0175b700 1 --2- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6ee803c9a0 0x7f6ee803ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:31.521 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.578+0000 7f6f0175b700 1 --2- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6ee803c9a0 0x7f6ee803ed90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8003a10 tx=0x7f6ef80092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:31.521 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.579+0000 7f6ef2ffd700 1 -- 172.21.15.129:0/3304565361 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f6ee803baf0 con 0x7f6ee803c9a0 2024-03-11T20:46:31.521 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.585+0000 7f6f041c0700 1 -- 172.21.15.129:0/3304565361 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f6efc063410 con 0x7f6ee803c9a0 2024-03-11T20:46:31.521 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.585+0000 7f6ef2ffd700 1 -- 172.21.15.129:0/3304565361 <== mgr.14152 v2:172.21.15.129:6800/2866184010 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f6efc063410 con 0x7f6ee803c9a0 2024-03-11T20:46:31.521 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.586+0000 7f6f041c0700 1 -- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6ee803c9a0 msgr2=0x7f6ee803ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:31.521 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.586+0000 7f6f041c0700 1 --2- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6ee803c9a0 0x7f6ee803ed90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8003a10 tx=0x7f6ef80092b0 comp rx=0 tx=0).stop 2024-03-11T20:46:31.521 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.586+0000 7f6f041c0700 1 -- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6efc07b720 msgr2=0x7f6efc19bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:31.521 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.586+0000 7f6f041c0700 1 --2- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6efc07b720 0x7f6efc19bba0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6eec003f60 tx=0x7f6eec0119b0 comp rx=0 tx=0).stop 2024-03-11T20:46:31.521 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.586+0000 7f6f041c0700 1 -- 172.21.15.129:0/3304565361 shutdown_connections 2024-03-11T20:46:31.521 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.586+0000 7f6f041c0700 1 --2- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6efc07b720 0x7f6efc19bba0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:31.522 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.586+0000 7f6f041c0700 1 --2- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6ee803c9a0 0x7f6ee803ed90 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:31.522 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.586+0000 7f6f041c0700 1 --2- 172.21.15.129:0/3304565361 >> [v2:172.21.15.129:6800/521270824,v1:172.21.15.129:6801/521270824] conn(0x7f6ee8038f20 0x7f6ee803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:31.522 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.586+0000 7f6f041c0700 1 -- 172.21.15.129:0/3304565361 >> 172.21.15.129:0/3304565361 conn(0x7f6efc1011c0 msgr2=0x7f6efc101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:31.522 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.586+0000 7f6f041c0700 1 -- 172.21.15.129:0/3304565361 shutdown_connections 2024-03-11T20:46:31.522 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:30.586+0000 7f6f041c0700 1 -- 172.21.15.129:0/3304565361 wait complete. 2024-03-11T20:46:31.522 INFO:teuthology.orchestra.run.smithi129.stdout:mgr epoch 9 is available 2024-03-11T20:46:31.522 INFO:teuthology.orchestra.run.smithi129.stdout:Generating a dashboard self-signed certificate... 2024-03-11T20:46:31.900 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:31 smithi129 ceph-mon[30184]: [11/Mar/2024:20:46:30] ENGINE Bus STARTING 2024-03-11T20:46:31.901 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:31 smithi129 ceph-mon[30184]: [11/Mar/2024:20:46:30] ENGINE Serving on http://172.21.15.129:8765 2024-03-11T20:46:31.901 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:31 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:32.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:32 smithi129 ceph-mon[30184]: [11/Mar/2024:20:46:30] ENGINE Serving on https://172.21.15.129:7150 2024-03-11T20:46:32.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:32 smithi129 ceph-mon[30184]: [11/Mar/2024:20:46:30] ENGINE Bus STARTED 2024-03-11T20:46:32.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:32 smithi129 ceph-mon[30184]: mgrmap e12: y(active, since 2s) 2024-03-11T20:46:32.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:32 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:32.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:32 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:32.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:32 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:32.987 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-11T20:46:32.987 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.013+0000 7fccac557700 1 Processor -- start 2024-03-11T20:46:32.987 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.013+0000 7fccac557700 1 -- start start 2024-03-11T20:46:32.987 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.014+0000 7fccac557700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcca41066a0 0x7fcca4108a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:32.987 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.014+0000 7fccac557700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca4078ea0 con 0x7fcca41066a0 2024-03-11T20:46:32.987 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.014+0000 7fccaa2f3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcca41066a0 0x7fcca4108a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:32.987 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.014+0000 7fccaa2f3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcca41066a0 0x7fcca4108a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57652/0 (socket says 172.21.15.129:57652) 2024-03-11T20:46:32.987 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.014+0000 7fccaa2f3700 1 -- 172.21.15.129:0/3983908051 learned_addr learned my addr 172.21.15.129:0/3983908051 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:32.987 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.014+0000 7fccaa2f3700 1 -- 172.21.15.129:0/3983908051 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca4108fd0 con 0x7fcca41066a0 2024-03-11T20:46:32.987 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.014+0000 7fccaa2f3700 1 --2- 172.21.15.129:0/3983908051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcca41066a0 0x7fcca4108a90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcca00098a0 tx=0x7fcca0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb913ea78b495a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:32.987 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.015+0000 7fcca92f1700 1 -- 172.21.15.129:0/3983908051 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcca0004160 con 0x7fcca41066a0 2024-03-11T20:46:32.987 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.015+0000 7fcca92f1700 1 -- 172.21.15.129:0/3983908051 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fcca0004300 con 0x7fcca41066a0 2024-03-11T20:46:32.988 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.015+0000 7fcca92f1700 1 -- 172.21.15.129:0/3983908051 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcca00118f0 con 0x7fcca41066a0 2024-03-11T20:46:32.988 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.015+0000 7fccac557700 1 -- 172.21.15.129:0/3983908051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcca41066a0 msgr2=0x7fcca4108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:32.988 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.015+0000 7fccac557700 1 --2- 172.21.15.129:0/3983908051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcca41066a0 0x7fcca4108a90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcca00098a0 tx=0x7fcca0009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:32.988 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.015+0000 7fccac557700 1 -- 172.21.15.129:0/3983908051 shutdown_connections 2024-03-11T20:46:32.988 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.015+0000 7fccac557700 1 --2- 172.21.15.129:0/3983908051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcca41066a0 0x7fcca4108a90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:32.988 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.015+0000 7fccac557700 1 -- 172.21.15.129:0/3983908051 >> 172.21.15.129:0/3983908051 conn(0x7fcca41000d0 msgr2=0x7fcca4102510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:32.988 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.015+0000 7fccac557700 1 -- 172.21.15.129:0/3983908051 shutdown_connections 2024-03-11T20:46:32.988 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.015+0000 7fccac557700 1 -- 172.21.15.129:0/3983908051 wait complete. 2024-03-11T20:46:32.988 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.016+0000 7fccac557700 1 Processor -- start 2024-03-11T20:46:32.988 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.016+0000 7fccac557700 1 -- start start 2024-03-11T20:46:32.988 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.016+0000 7fccac557700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcca41066a0 0x7fcca41999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:32.988 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.016+0000 7fccac557700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca4109f20 con 0x7fcca41066a0 2024-03-11T20:46:32.988 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.016+0000 7fccaa2f3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcca41066a0 0x7fcca41999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:32.988 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.016+0000 7fccaa2f3700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcca41066a0 0x7fcca41999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57664/0 (socket says 172.21.15.129:57664) 2024-03-11T20:46:32.989 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.017+0000 7fccaa2f3700 1 -- 172.21.15.129:0/390851931 learned_addr learned my addr 172.21.15.129:0/390851931 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:32.989 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.017+0000 7fccaa2f3700 1 -- 172.21.15.129:0/390851931 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca4199ee0 con 0x7fcca41066a0 2024-03-11T20:46:32.989 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.017+0000 7fccaa2f3700 1 --2- 172.21.15.129:0/390851931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcca41066a0 0x7fcca41999a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcca0011970 tx=0x7fcca0011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:32.989 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.017+0000 7fcc9b7fe700 1 -- 172.21.15.129:0/390851931 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcca001a430 con 0x7fcca41066a0 2024-03-11T20:46:32.989 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.017+0000 7fcc9b7fe700 1 -- 172.21.15.129:0/390851931 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fcca001aa50 con 0x7fcca41066a0 2024-03-11T20:46:32.989 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.017+0000 7fccac557700 1 -- 172.21.15.129:0/390851931 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca419a170 con 0x7fcca41066a0 2024-03-11T20:46:32.989 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.017+0000 7fcc9b7fe700 1 -- 172.21.15.129:0/390851931 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcca0019910 con 0x7fcca41066a0 2024-03-11T20:46:32.989 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.017+0000 7fccac557700 1 -- 172.21.15.129:0/390851931 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca419ce60 con 0x7fcca41066a0 2024-03-11T20:46:32.989 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.018+0000 7fcc9b7fe700 1 -- 172.21.15.129:0/390851931 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7fcca0022020 con 0x7fcca41066a0 2024-03-11T20:46:32.989 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.018+0000 7fcc9b7fe700 1 --2- 172.21.15.129:0/390851931 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fcc90038e60 0x7fcc9003b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:32.989 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.018+0000 7fcc9b7fe700 1 -- 172.21.15.129:0/390851931 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcca001a5d0 con 0x7fcca41066a0 2024-03-11T20:46:32.989 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.018+0000 7fcca9af2700 1 --2- 172.21.15.129:0/390851931 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fcc90038e60 0x7fcc9003b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:32.989 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.018+0000 7fccac557700 1 -- 172.21.15.129:0/390851931 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca4063410 con 0x7fcca41066a0 2024-03-11T20:46:32.989 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.019+0000 7fcca9af2700 1 --2- 172.21.15.129:0/390851931 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fcc90038e60 0x7fcc9003b320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcc94006fd0 tx=0x7fcc94006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:32.990 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.023+0000 7fcc9b7fe700 1 -- 172.21.15.129:0/390851931 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fcca002c8c0 con 0x7fcca41066a0 2024-03-11T20:46:32.990 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.191+0000 7fcc9b7fe700 1 -- 172.21.15.129:0/390851931 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fcca002c3a0 con 0x7fcca41066a0 2024-03-11T20:46:32.990 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.228+0000 7fccac557700 1 -- 172.21.15.129:0/390851931 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fcca41012b0 con 0x7fcc90038e60 2024-03-11T20:46:32.990 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.371+0000 7fcc9b7fe700 1 -- 172.21.15.129:0/390851931 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fcca41012b0 con 0x7fcc90038e60 2024-03-11T20:46:32.990 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.373+0000 7fccac557700 1 -- 172.21.15.129:0/390851931 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fcc90038e60 msgr2=0x7fcc9003b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:32.990 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.373+0000 7fccac557700 1 --2- 172.21.15.129:0/390851931 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fcc90038e60 0x7fcc9003b320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcc94006fd0 tx=0x7fcc94006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:32.990 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.373+0000 7fccac557700 1 -- 172.21.15.129:0/390851931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcca41066a0 msgr2=0x7fcca41999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:32.990 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.373+0000 7fccac557700 1 --2- 172.21.15.129:0/390851931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcca41066a0 0x7fcca41999a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcca0011970 tx=0x7fcca0011a50 comp rx=0 tx=0).stop 2024-03-11T20:46:32.990 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.373+0000 7fccac557700 1 -- 172.21.15.129:0/390851931 shutdown_connections 2024-03-11T20:46:32.990 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.373+0000 7fccac557700 1 --2- 172.21.15.129:0/390851931 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fcc90038e60 0x7fcc9003b320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:32.990 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.373+0000 7fccac557700 1 --2- 172.21.15.129:0/390851931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcca41066a0 0x7fcca41999a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:32.990 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.373+0000 7fccac557700 1 -- 172.21.15.129:0/390851931 >> 172.21.15.129:0/390851931 conn(0x7fcca41000d0 msgr2=0x7fcca4100ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:32.990 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.373+0000 7fccac557700 1 -- 172.21.15.129:0/390851931 shutdown_connections 2024-03-11T20:46:32.991 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:32.373+0000 7fccac557700 1 -- 172.21.15.129:0/390851931 wait complete. 2024-03-11T20:46:32.991 INFO:teuthology.orchestra.run.smithi129.stdout:Creating initial admin user... 2024-03-11T20:46:33.920 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:33 smithi129 ceph-mon[30184]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:46:34.708 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:34 smithi129 ceph-mon[30184]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:46:34.708 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:34 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:34.715 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$SNIh8fMizvNgay3x/.b7uuKnAz7a6SxH8HIguvXrjgFbIFPhXdh.S", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710189993, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T20:46:34.715 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.463+0000 7f07d4aa0700 1 Processor -- start 2024-03-11T20:46:34.715 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.463+0000 7f07d4aa0700 1 -- start start 2024-03-11T20:46:34.716 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.463+0000 7f07d4aa0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07d0106700 0x7f07d0108af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:34.716 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.463+0000 7f07d4aa0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d00796d0 con 0x7f07d0106700 2024-03-11T20:46:34.716 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.464+0000 7f07ce59c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07d0106700 0x7f07d0108af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:34.716 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.464+0000 7f07ce59c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07d0106700 0x7f07d0108af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57670/0 (socket says 172.21.15.129:57670) 2024-03-11T20:46:34.716 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.464+0000 7f07ce59c700 1 -- 172.21.15.129:0/880055035 learned_addr learned my addr 172.21.15.129:0/880055035 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:34.716 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.464+0000 7f07ce59c700 1 -- 172.21.15.129:0/880055035 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d0109030 con 0x7f07d0106700 2024-03-11T20:46:34.716 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.464+0000 7f07ce59c700 1 --2- 172.21.15.129:0/880055035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07d0106700 0x7f07d0108af0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f07c00098a0 tx=0x7f07c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f785fa351bc7425a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:34.716 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.464+0000 7f07cd59a700 1 -- 172.21.15.129:0/880055035 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07c0004160 con 0x7f07d0106700 2024-03-11T20:46:34.716 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.464+0000 7f07cd59a700 1 -- 172.21.15.129:0/880055035 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f07c0004300 con 0x7f07d0106700 2024-03-11T20:46:34.716 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.464+0000 7f07cd59a700 1 -- 172.21.15.129:0/880055035 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07c00118f0 con 0x7f07d0106700 2024-03-11T20:46:34.716 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.465+0000 7f07d4aa0700 1 -- 172.21.15.129:0/880055035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07d0106700 msgr2=0x7f07d0108af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:34.716 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.465+0000 7f07d4aa0700 1 --2- 172.21.15.129:0/880055035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07d0106700 0x7f07d0108af0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f07c00098a0 tx=0x7f07c0009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:34.716 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.465+0000 7f07d4aa0700 1 -- 172.21.15.129:0/880055035 shutdown_connections 2024-03-11T20:46:34.717 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.465+0000 7f07d4aa0700 1 --2- 172.21.15.129:0/880055035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07d0106700 0x7f07d0108af0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:34.717 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.465+0000 7f07d4aa0700 1 -- 172.21.15.129:0/880055035 >> 172.21.15.129:0/880055035 conn(0x7f07d01000b0 msgr2=0x7f07d0102510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:34.717 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.466+0000 7f07d4aa0700 1 -- 172.21.15.129:0/880055035 shutdown_connections 2024-03-11T20:46:34.717 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.466+0000 7f07d4aa0700 1 -- 172.21.15.129:0/880055035 wait complete. 2024-03-11T20:46:34.717 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.466+0000 7f07d4aa0700 1 Processor -- start 2024-03-11T20:46:34.717 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.466+0000 7f07d4aa0700 1 -- start start 2024-03-11T20:46:34.717 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.467+0000 7f07d4aa0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07d0106700 0x7f07d0199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:34.717 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.467+0000 7f07d4aa0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d0109f20 con 0x7f07d0106700 2024-03-11T20:46:34.717 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.467+0000 7f07ce59c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07d0106700 0x7f07d0199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:34.717 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.467+0000 7f07ce59c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07d0106700 0x7f07d0199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57674/0 (socket says 172.21.15.129:57674) 2024-03-11T20:46:34.717 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.467+0000 7f07ce59c700 1 -- 172.21.15.129:0/1630639313 learned_addr learned my addr 172.21.15.129:0/1630639313 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:34.717 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.467+0000 7f07ce59c700 1 -- 172.21.15.129:0/1630639313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d0199e80 con 0x7f07d0106700 2024-03-11T20:46:34.717 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.467+0000 7f07ce59c700 1 --2- 172.21.15.129:0/1630639313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07d0106700 0x7f07d0199940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f07c0003f60 tx=0x7f07c0011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:34.717 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.467+0000 7f07bf7fe700 1 -- 172.21.15.129:0/1630639313 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07c00038b0 con 0x7f07d0106700 2024-03-11T20:46:34.718 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.468+0000 7f07d4aa0700 1 -- 172.21.15.129:0/1630639313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d019a110 con 0x7f07d0106700 2024-03-11T20:46:34.718 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.468+0000 7f07bf7fe700 1 -- 172.21.15.129:0/1630639313 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f07c0003a50 con 0x7f07d0106700 2024-03-11T20:46:34.718 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.468+0000 7f07bf7fe700 1 -- 172.21.15.129:0/1630639313 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07c0024540 con 0x7f07d0106700 2024-03-11T20:46:34.718 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.468+0000 7f07d4aa0700 1 -- 172.21.15.129:0/1630639313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d019a530 con 0x7f07d0106700 2024-03-11T20:46:34.718 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.469+0000 7f07bf7fe700 1 -- 172.21.15.129:0/1630639313 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f07c0022020 con 0x7f07d0106700 2024-03-11T20:46:34.718 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.469+0000 7f07d4aa0700 1 -- 172.21.15.129:0/1630639313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d0063410 con 0x7f07d0106700 2024-03-11T20:46:34.718 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.469+0000 7f07bf7fe700 1 --2- 172.21.15.129:0/1630639313 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f07b8038f70 0x7f07b803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:34.718 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.469+0000 7f07bf7fe700 1 -- 172.21.15.129:0/1630639313 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f07c004cfd0 con 0x7f07d0106700 2024-03-11T20:46:34.718 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.469+0000 7f07cdd9b700 1 --2- 172.21.15.129:0/1630639313 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f07b8038f70 0x7f07b803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:34.718 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.469+0000 7f07cdd9b700 1 --2- 172.21.15.129:0/1630639313 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f07b8038f70 0x7f07b803b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f07c4006fd0 tx=0x7f07c4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:34.718 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.475+0000 7f07bf7fe700 1 -- 172.21.15.129:0/1630639313 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f07c0017ca0 con 0x7f07d0106700 2024-03-11T20:46:34.718 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.698+0000 7f07d4aa0700 1 -- 172.21.15.129:0/1630639313 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f07d019cf20 con 0x7f07b8038f70 2024-03-11T20:46:34.718 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.925+0000 7f07bf7fe700 1 -- 172.21.15.129:0/1630639313 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f07d019cf20 con 0x7f07b8038f70 2024-03-11T20:46:34.719 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.928+0000 7f07d4aa0700 1 -- 172.21.15.129:0/1630639313 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f07b8038f70 msgr2=0x7f07b803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:34.719 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.928+0000 7f07d4aa0700 1 --2- 172.21.15.129:0/1630639313 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f07b8038f70 0x7f07b803b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f07c4006fd0 tx=0x7f07c4006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:34.719 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.928+0000 7f07d4aa0700 1 -- 172.21.15.129:0/1630639313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07d0106700 msgr2=0x7f07d0199940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:34.719 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.928+0000 7f07d4aa0700 1 --2- 172.21.15.129:0/1630639313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07d0106700 0x7f07d0199940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f07c0003f60 tx=0x7f07c0011ef0 comp rx=0 tx=0).stop 2024-03-11T20:46:34.719 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.928+0000 7f07d4aa0700 1 -- 172.21.15.129:0/1630639313 shutdown_connections 2024-03-11T20:46:34.719 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.928+0000 7f07d4aa0700 1 --2- 172.21.15.129:0/1630639313 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f07b8038f70 0x7f07b803b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:34.719 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.928+0000 7f07d4aa0700 1 --2- 172.21.15.129:0/1630639313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07d0106700 0x7f07d0199940 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:34.719 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.928+0000 7f07d4aa0700 1 -- 172.21.15.129:0/1630639313 >> 172.21.15.129:0/1630639313 conn(0x7f07d01000b0 msgr2=0x7f07d0100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:34.719 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.928+0000 7f07d4aa0700 1 -- 172.21.15.129:0/1630639313 shutdown_connections 2024-03-11T20:46:34.719 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:33.928+0000 7f07d4aa0700 1 -- 172.21.15.129:0/1630639313 wait complete. 2024-03-11T20:46:34.719 INFO:teuthology.orchestra.run.smithi129.stdout:Fetching dashboard port number... 2024-03-11T20:46:36.608 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stdout 8443 2024-03-11T20:46:36.608 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.632+0000 7f2b1d8a7700 1 Processor -- start 2024-03-11T20:46:36.608 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.632+0000 7f2b1d8a7700 1 -- start start 2024-03-11T20:46:36.608 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.632+0000 7f2b1d8a7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b18105c70 0x7f2b18106090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:36.608 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.632+0000 7f2b1d8a7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b18106660 con 0x7f2b18105c70 2024-03-11T20:46:36.608 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.632+0000 7f2b16ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b18105c70 0x7f2b18106090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:36.608 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.632+0000 7f2b16ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b18105c70 0x7f2b18106090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47554/0 (socket says 172.21.15.129:47554) 2024-03-11T20:46:36.608 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.632+0000 7f2b16ffd700 1 -- 172.21.15.129:0/3655722057 learned_addr learned my addr 172.21.15.129:0/3655722057 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:36.608 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.632+0000 7f2b16ffd700 1 -- 172.21.15.129:0/3655722057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b18106e70 con 0x7f2b18105c70 2024-03-11T20:46:36.608 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.632+0000 7f2b16ffd700 1 --2- 172.21.15.129:0/3655722057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b18105c70 0x7f2b18106090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2b080098a0 tx=0x7f2b08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2996e326bbdb0a96 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:36.608 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.633+0000 7f2b15ffb700 1 -- 172.21.15.129:0/3655722057 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b08004160 con 0x7f2b18105c70 2024-03-11T20:46:36.608 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.633+0000 7f2b15ffb700 1 -- 172.21.15.129:0/3655722057 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2b08004300 con 0x7f2b18105c70 2024-03-11T20:46:36.609 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.633+0000 7f2b15ffb700 1 -- 172.21.15.129:0/3655722057 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b080118f0 con 0x7f2b18105c70 2024-03-11T20:46:36.609 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.633+0000 7f2b1d8a7700 1 -- 172.21.15.129:0/3655722057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b18105c70 msgr2=0x7f2b18106090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:36.609 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.633+0000 7f2b1d8a7700 1 --2- 172.21.15.129:0/3655722057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b18105c70 0x7f2b18106090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2b080098a0 tx=0x7f2b08009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:36.609 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.633+0000 7f2b1d8a7700 1 -- 172.21.15.129:0/3655722057 shutdown_connections 2024-03-11T20:46:36.609 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.633+0000 7f2b1d8a7700 1 --2- 172.21.15.129:0/3655722057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b18105c70 0x7f2b18106090 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:36.609 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.633+0000 7f2b1d8a7700 1 -- 172.21.15.129:0/3655722057 >> 172.21.15.129:0/3655722057 conn(0x7f2b18101250 msgr2=0x7f2b18103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:36.609 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.633+0000 7f2b1d8a7700 1 -- 172.21.15.129:0/3655722057 shutdown_connections 2024-03-11T20:46:36.609 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.633+0000 7f2b1d8a7700 1 -- 172.21.15.129:0/3655722057 wait complete. 2024-03-11T20:46:36.609 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.634+0000 7f2b1d8a7700 1 Processor -- start 2024-03-11T20:46:36.609 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.634+0000 7f2b1d8a7700 1 -- start start 2024-03-11T20:46:36.609 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.634+0000 7f2b1d8a7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b18105c70 0x7f2b181999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:36.609 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.634+0000 7f2b1d8a7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1810a0a0 con 0x7f2b18105c70 2024-03-11T20:46:36.609 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.634+0000 7f2b16ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b18105c70 0x7f2b181999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:36.609 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.634+0000 7f2b16ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b18105c70 0x7f2b181999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47564/0 (socket says 172.21.15.129:47564) 2024-03-11T20:46:36.610 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.634+0000 7f2b16ffd700 1 -- 172.21.15.129:0/3581381585 learned_addr learned my addr 172.21.15.129:0/3581381585 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:36.610 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.635+0000 7f2b16ffd700 1 -- 172.21.15.129:0/3581381585 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b18199f00 con 0x7f2b18105c70 2024-03-11T20:46:36.610 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.635+0000 7f2b16ffd700 1 --2- 172.21.15.129:0/3581381585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b18105c70 0x7f2b181999c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2b08003f60 tx=0x7f2b08011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:36.610 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.635+0000 7f2b1c8a5700 1 -- 172.21.15.129:0/3581381585 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b080038b0 con 0x7f2b18105c70 2024-03-11T20:46:36.610 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.635+0000 7f2b1c8a5700 1 -- 172.21.15.129:0/3581381585 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2b08003a50 con 0x7f2b18105c70 2024-03-11T20:46:36.610 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.635+0000 7f2b1d8a7700 1 -- 172.21.15.129:0/3581381585 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1819a190 con 0x7f2b18105c70 2024-03-11T20:46:36.610 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.635+0000 7f2b1c8a5700 1 -- 172.21.15.129:0/3581381585 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b08024410 con 0x7f2b18105c70 2024-03-11T20:46:36.610 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.635+0000 7f2b1d8a7700 1 -- 172.21.15.129:0/3581381585 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1819ce80 con 0x7f2b18105c70 2024-03-11T20:46:36.610 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.636+0000 7f2b1c8a5700 1 -- 172.21.15.129:0/3581381585 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f2b08022020 con 0x7f2b18105c70 2024-03-11T20:46:36.610 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.636+0000 7f2b1c8a5700 1 --2- 172.21.15.129:0/3581381585 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2b00039070 0x7f2b0003b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:36.610 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.636+0000 7f2b1c8a5700 1 -- 172.21.15.129:0/3581381585 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2b0801a430 con 0x7f2b18105c70 2024-03-11T20:46:36.610 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.636+0000 7f2b167fc700 1 --2- 172.21.15.129:0/3581381585 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2b00039070 0x7f2b0003b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:36.610 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.636+0000 7f2b1d8a7700 1 -- 172.21.15.129:0/3581381585 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b18063410 con 0x7f2b18105c70 2024-03-11T20:46:36.611 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.636+0000 7f2b167fc700 1 --2- 172.21.15.129:0/3581381585 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2b00039070 0x7f2b0003b530 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c006fd0 tx=0x7f2b0c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:36.611 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.641+0000 7f2b1c8a5700 1 -- 172.21.15.129:0/3581381585 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f2b0800b3e0 con 0x7f2b18105c70 2024-03-11T20:46:36.611 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.835+0000 7f2b1d8a7700 1 -- 172.21.15.129:0/3581381585 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f2b1819d120 con 0x7f2b18105c70 2024-03-11T20:46:36.611 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.835+0000 7f2b1c8a5700 1 -- 172.21.15.129:0/3581381585 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f2b080172b0 con 0x7f2b18105c70 2024-03-11T20:46:36.611 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.837+0000 7f2b1d8a7700 1 -- 172.21.15.129:0/3581381585 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2b00039070 msgr2=0x7f2b0003b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:36.611 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.837+0000 7f2b1d8a7700 1 --2- 172.21.15.129:0/3581381585 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2b00039070 0x7f2b0003b530 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c006fd0 tx=0x7f2b0c006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:36.611 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.837+0000 7f2b1d8a7700 1 -- 172.21.15.129:0/3581381585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b18105c70 msgr2=0x7f2b181999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:36.611 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.837+0000 7f2b1d8a7700 1 --2- 172.21.15.129:0/3581381585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b18105c70 0x7f2b181999c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2b08003f60 tx=0x7f2b08011ef0 comp rx=0 tx=0).stop 2024-03-11T20:46:36.611 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.837+0000 7f2b1d8a7700 1 -- 172.21.15.129:0/3581381585 shutdown_connections 2024-03-11T20:46:36.611 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.837+0000 7f2b1d8a7700 1 --2- 172.21.15.129:0/3581381585 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2b00039070 0x7f2b0003b530 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:36.611 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.837+0000 7f2b1d8a7700 1 --2- 172.21.15.129:0/3581381585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b18105c70 0x7f2b181999c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:36.611 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.837+0000 7f2b1d8a7700 1 -- 172.21.15.129:0/3581381585 >> 172.21.15.129:0/3581381585 conn(0x7f2b18101250 msgr2=0x7f2b18101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:36.611 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.837+0000 7f2b1d8a7700 1 -- 172.21.15.129:0/3581381585 shutdown_connections 2024-03-11T20:46:36.611 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:35.837+0000 7f2b1d8a7700 1 -- 172.21.15.129:0/3581381585 wait complete. 2024-03-11T20:46:36.614 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T20:46:36.614 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout disabled 2024-03-11T20:46:36.621 INFO:teuthology.orchestra.run.smithi129.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T20:46:36.621 INFO:teuthology.orchestra.run.smithi129.stdout:systemctl: stdout inactive 2024-03-11T20:46:36.621 INFO:teuthology.orchestra.run.smithi129.stdout:firewalld.service is not enabled 2024-03-11T20:46:36.621 INFO:teuthology.orchestra.run.smithi129.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T20:46:36.623 INFO:teuthology.orchestra.run.smithi129.stdout:Ceph Dashboard is now available at: 2024-03-11T20:46:36.623 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:46:36.623 INFO:teuthology.orchestra.run.smithi129.stdout: URL: https://smithi129.front.sepia.ceph.com:8443/ 2024-03-11T20:46:36.623 INFO:teuthology.orchestra.run.smithi129.stdout: User: admin 2024-03-11T20:46:36.623 INFO:teuthology.orchestra.run.smithi129.stdout: Password: 759ztccru0 2024-03-11T20:46:36.623 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:46:36.624 INFO:teuthology.orchestra.run.smithi129.stdout:Saving cluster configuration to /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config directory 2024-03-11T20:46:36.624 INFO:teuthology.orchestra.run.smithi129.stdout:Enabling autotune for osd_memory_target 2024-03-11T20:46:36.856 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:36 smithi129 ceph-mon[30184]: mgrmap e13: y(active, since 6s) 2024-03-11T20:46:36.856 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3581381585' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T20:46:37.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.094+0000 7fa90dca0700 1 Processor -- start 2024-03-11T20:46:37.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.094+0000 7fa90dca0700 1 -- start start 2024-03-11T20:46:37.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.094+0000 7fa90dca0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa908105c70 0x7fa908106090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:37.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.094+0000 7fa90dca0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa908106660 con 0x7fa908105c70 2024-03-11T20:46:37.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.095+0000 7fa9077fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa908105c70 0x7fa908106090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:37.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.095+0000 7fa9077fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa908105c70 0x7fa908106090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47580/0 (socket says 172.21.15.129:47580) 2024-03-11T20:46:37.872 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.095+0000 7fa9077fe700 1 -- 172.21.15.129:0/3062795027 learned_addr learned my addr 172.21.15.129:0/3062795027 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:37.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.095+0000 7fa9077fe700 1 -- 172.21.15.129:0/3062795027 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa908106e70 con 0x7fa908105c70 2024-03-11T20:46:37.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.095+0000 7fa9077fe700 1 --2- 172.21.15.129:0/3062795027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa908105c70 0x7fa908106090 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa8f80098a0 tx=0x7fa8f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d3015b04b015f2d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:37.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.095+0000 7fa9067fc700 1 -- 172.21.15.129:0/3062795027 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8f8004160 con 0x7fa908105c70 2024-03-11T20:46:37.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.095+0000 7fa9067fc700 1 -- 172.21.15.129:0/3062795027 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa8f8004300 con 0x7fa908105c70 2024-03-11T20:46:37.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.095+0000 7fa9067fc700 1 -- 172.21.15.129:0/3062795027 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8f80118f0 con 0x7fa908105c70 2024-03-11T20:46:37.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.096+0000 7fa90dca0700 1 -- 172.21.15.129:0/3062795027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa908105c70 msgr2=0x7fa908106090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:37.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.096+0000 7fa90dca0700 1 --2- 172.21.15.129:0/3062795027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa908105c70 0x7fa908106090 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa8f80098a0 tx=0x7fa8f8009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:37.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.096+0000 7fa90dca0700 1 -- 172.21.15.129:0/3062795027 shutdown_connections 2024-03-11T20:46:37.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.096+0000 7fa90dca0700 1 --2- 172.21.15.129:0/3062795027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa908105c70 0x7fa908106090 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:37.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.096+0000 7fa90dca0700 1 -- 172.21.15.129:0/3062795027 >> 172.21.15.129:0/3062795027 conn(0x7fa9081011f0 msgr2=0x7fa908103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:37.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.096+0000 7fa90dca0700 1 -- 172.21.15.129:0/3062795027 shutdown_connections 2024-03-11T20:46:37.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.096+0000 7fa90dca0700 1 -- 172.21.15.129:0/3062795027 wait complete. 2024-03-11T20:46:37.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.096+0000 7fa90dca0700 1 Processor -- start 2024-03-11T20:46:37.873 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.097+0000 7fa90dca0700 1 -- start start 2024-03-11T20:46:37.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.097+0000 7fa90dca0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa908105c70 0x7fa90807b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:37.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.097+0000 7fa90dca0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90810a0a0 con 0x7fa908105c70 2024-03-11T20:46:37.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.097+0000 7fa9077fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa908105c70 0x7fa90807b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:37.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.097+0000 7fa9077fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa908105c70 0x7fa90807b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47596/0 (socket says 172.21.15.129:47596) 2024-03-11T20:46:37.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.097+0000 7fa9077fe700 1 -- 172.21.15.129:0/3516777793 learned_addr learned my addr 172.21.15.129:0/3516777793 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:37.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.097+0000 7fa9077fe700 1 -- 172.21.15.129:0/3516777793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90807b930 con 0x7fa908105c70 2024-03-11T20:46:37.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.097+0000 7fa9077fe700 1 --2- 172.21.15.129:0/3516777793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa908105c70 0x7fa90807b3f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8003f60 tx=0x7fa8f8011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:37.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.097+0000 7fa904ff9700 1 -- 172.21.15.129:0/3516777793 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8f80038b0 con 0x7fa908105c70 2024-03-11T20:46:37.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.097+0000 7fa904ff9700 1 -- 172.21.15.129:0/3516777793 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa8f8003a50 con 0x7fa908105c70 2024-03-11T20:46:37.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.097+0000 7fa90dca0700 1 -- 172.21.15.129:0/3516777793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa908079b20 con 0x7fa908105c70 2024-03-11T20:46:37.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.097+0000 7fa904ff9700 1 -- 172.21.15.129:0/3516777793 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8f8024410 con 0x7fa908105c70 2024-03-11T20:46:37.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.097+0000 7fa90dca0700 1 -- 172.21.15.129:0/3516777793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90807a060 con 0x7fa908105c70 2024-03-11T20:46:37.874 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.098+0000 7fa904ff9700 1 -- 172.21.15.129:0/3516777793 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fa8f8022020 con 0x7fa908105c70 2024-03-11T20:46:37.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.098+0000 7fa90dca0700 1 -- 172.21.15.129:0/3516777793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa908063410 con 0x7fa908105c70 2024-03-11T20:46:37.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.099+0000 7fa904ff9700 1 --2- 172.21.15.129:0/3516777793 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fa8f0039070 0x7fa8f003b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:37.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.099+0000 7fa904ff9700 1 -- 172.21.15.129:0/3516777793 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa8f804cf70 con 0x7fa908105c70 2024-03-11T20:46:37.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.099+0000 7fa906ffd700 1 --2- 172.21.15.129:0/3516777793 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fa8f0039070 0x7fa8f003b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:37.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.099+0000 7fa906ffd700 1 --2- 172.21.15.129:0/3516777793 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fa8f0039070 0x7fa8f003b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc006fd0 tx=0x7fa8fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:37.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.103+0000 7fa904ff9700 1 -- 172.21.15.129:0/3516777793 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fa8f800b3e0 con 0x7fa908105c70 2024-03-11T20:46:37.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.295+0000 7fa90dca0700 1 -- 172.21.15.129:0/3516777793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fa90807a7c0 con 0x7fa908105c70 2024-03-11T20:46:37.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.296+0000 7fa904ff9700 1 -- 172.21.15.129:0/3516777793 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fa8f802dc90 con 0x7fa908105c70 2024-03-11T20:46:37.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.298+0000 7fa90dca0700 1 -- 172.21.15.129:0/3516777793 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fa8f0039070 msgr2=0x7fa8f003b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:37.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.298+0000 7fa90dca0700 1 --2- 172.21.15.129:0/3516777793 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fa8f0039070 0x7fa8f003b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc006fd0 tx=0x7fa8fc006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:37.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.298+0000 7fa90dca0700 1 -- 172.21.15.129:0/3516777793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa908105c70 msgr2=0x7fa90807b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:37.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.298+0000 7fa90dca0700 1 --2- 172.21.15.129:0/3516777793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa908105c70 0x7fa90807b3f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8003f60 tx=0x7fa8f8011ef0 comp rx=0 tx=0).stop 2024-03-11T20:46:37.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.298+0000 7fa90dca0700 1 -- 172.21.15.129:0/3516777793 shutdown_connections 2024-03-11T20:46:37.875 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.298+0000 7fa90dca0700 1 --2- 172.21.15.129:0/3516777793 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fa8f0039070 0x7fa8f003b530 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:37.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.298+0000 7fa90dca0700 1 --2- 172.21.15.129:0/3516777793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa908105c70 0x7fa90807b3f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:37.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.298+0000 7fa90dca0700 1 -- 172.21.15.129:0/3516777793 >> 172.21.15.129:0/3516777793 conn(0x7fa9081011f0 msgr2=0x7fa908101cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:37.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.299+0000 7fa90dca0700 1 -- 172.21.15.129:0/3516777793 shutdown_connections 2024-03-11T20:46:37.876 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:37.299+0000 7fa90dca0700 1 -- 172.21.15.129:0/3516777793 wait complete. 2024-03-11T20:46:39.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.317+0000 7f207f45d700 1 Processor -- start 2024-03-11T20:46:39.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.317+0000 7f207f45d700 1 -- start start 2024-03-11T20:46:39.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.317+0000 7f207f45d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078105c60 0x7f2078106080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:39.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.317+0000 7f207f45d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2078106650 con 0x7f2078105c60 2024-03-11T20:46:39.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.318+0000 7f207d1f9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078105c60 0x7f2078106080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:39.187 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.318+0000 7f207d1f9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078105c60 0x7f2078106080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47612/0 (socket says 172.21.15.129:47612) 2024-03-11T20:46:39.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.318+0000 7f207d1f9700 1 -- 172.21.15.129:0/247758378 learned_addr learned my addr 172.21.15.129:0/247758378 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:39.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.318+0000 7f207d1f9700 1 -- 172.21.15.129:0/247758378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2078106e60 con 0x7f2078105c60 2024-03-11T20:46:39.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.318+0000 7f207d1f9700 1 --2- 172.21.15.129:0/247758378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078105c60 0x7f2078106080 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f20740098a0 tx=0x7f2074009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d75c6a4a8b1e882 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:39.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.318+0000 7f206ffff700 1 -- 172.21.15.129:0/247758378 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2074004160 con 0x7f2078105c60 2024-03-11T20:46:39.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.318+0000 7f206ffff700 1 -- 172.21.15.129:0/247758378 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2074004300 con 0x7f2078105c60 2024-03-11T20:46:39.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.318+0000 7f206ffff700 1 -- 172.21.15.129:0/247758378 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20740118f0 con 0x7f2078105c60 2024-03-11T20:46:39.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.319+0000 7f207f45d700 1 -- 172.21.15.129:0/247758378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078105c60 msgr2=0x7f2078106080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:39.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.319+0000 7f207f45d700 1 --2- 172.21.15.129:0/247758378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078105c60 0x7f2078106080 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f20740098a0 tx=0x7f2074009bb0 comp rx=0 tx=0).stop 2024-03-11T20:46:39.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.319+0000 7f207f45d700 1 -- 172.21.15.129:0/247758378 shutdown_connections 2024-03-11T20:46:39.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.319+0000 7f207f45d700 1 --2- 172.21.15.129:0/247758378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078105c60 0x7f2078106080 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:39.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.319+0000 7f207f45d700 1 -- 172.21.15.129:0/247758378 >> 172.21.15.129:0/247758378 conn(0x7f20781011c0 msgr2=0x7f2078103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:39.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.319+0000 7f207f45d700 1 -- 172.21.15.129:0/247758378 shutdown_connections 2024-03-11T20:46:39.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.319+0000 7f207f45d700 1 -- 172.21.15.129:0/247758378 wait complete. 2024-03-11T20:46:39.188 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.319+0000 7f207f45d700 1 Processor -- start 2024-03-11T20:46:39.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.319+0000 7f207f45d700 1 -- start start 2024-03-11T20:46:39.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.319+0000 7f207f45d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078105c60 0x7f207819bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:39.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.320+0000 7f207f45d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f207810a090 con 0x7f2078105c60 2024-03-11T20:46:39.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.320+0000 7f207d1f9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078105c60 0x7f207819bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:39.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.320+0000 7f207d1f9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078105c60 0x7f207819bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47624/0 (socket says 172.21.15.129:47624) 2024-03-11T20:46:39.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.320+0000 7f207d1f9700 1 -- 172.21.15.129:0/1713423158 learned_addr learned my addr 172.21.15.129:0/1713423158 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:39.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.320+0000 7f207d1f9700 1 -- 172.21.15.129:0/1713423158 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207819c1d0 con 0x7f2078105c60 2024-03-11T20:46:39.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.320+0000 7f207d1f9700 1 --2- 172.21.15.129:0/1713423158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078105c60 0x7f207819bc90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2074009fb0 tx=0x7f2074011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:39.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.320+0000 7f206e7fc700 1 -- 172.21.15.129:0/1713423158 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f207401a430 con 0x7f2078105c60 2024-03-11T20:46:39.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.320+0000 7f207f45d700 1 -- 172.21.15.129:0/1713423158 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f207819c460 con 0x7f2078105c60 2024-03-11T20:46:39.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.320+0000 7f206e7fc700 1 -- 172.21.15.129:0/1713423158 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f207401aa50 con 0x7f2078105c60 2024-03-11T20:46:39.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.320+0000 7f206e7fc700 1 -- 172.21.15.129:0/1713423158 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2074019800 con 0x7f2078105c60 2024-03-11T20:46:39.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.320+0000 7f207f45d700 1 -- 172.21.15.129:0/1713423158 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f207819f150 con 0x7f2078105c60 2024-03-11T20:46:39.189 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.321+0000 7f206e7fc700 1 -- 172.21.15.129:0/1713423158 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f2074022020 con 0x7f2078105c60 2024-03-11T20:46:39.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.321+0000 7f206e7fc700 1 --2- 172.21.15.129:0/1713423158 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2064038fc0 0x7f206403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:39.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.321+0000 7f206e7fc700 1 -- 172.21.15.129:0/1713423158 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f207404d540 con 0x7f2078105c60 2024-03-11T20:46:39.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.321+0000 7f207f45d700 1 -- 172.21.15.129:0/1713423158 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2078063410 con 0x7f2078105c60 2024-03-11T20:46:39.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.321+0000 7f207c9f8700 1 --2- 172.21.15.129:0/1713423158 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2064038fc0 0x7f206403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:39.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.322+0000 7f207c9f8700 1 --2- 172.21.15.129:0/1713423158 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2064038fc0 0x7f206403b480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2068006fd0 tx=0x7f2068006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:39.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.326+0000 7f206e7fc700 1 -- 172.21.15.129:0/1713423158 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f2074004300 con 0x7f2078105c60 2024-03-11T20:46:39.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.631+0000 7f207f45d700 1 -- 172.21.15.129:0/1713423158 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f207819f470 con 0x7f2078105c60 2024-03-11T20:46:39.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.633+0000 7f206e7fc700 1 -- 172.21.15.129:0/1713423158 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f2074020c10 con 0x7f2078105c60 2024-03-11T20:46:39.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.635+0000 7f207f45d700 1 -- 172.21.15.129:0/1713423158 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2064038fc0 msgr2=0x7f206403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:39.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.635+0000 7f207f45d700 1 --2- 172.21.15.129:0/1713423158 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2064038fc0 0x7f206403b480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2068006fd0 tx=0x7f2068006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:39.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.635+0000 7f207f45d700 1 -- 172.21.15.129:0/1713423158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078105c60 msgr2=0x7f207819bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:39.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.635+0000 7f207f45d700 1 --2- 172.21.15.129:0/1713423158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078105c60 0x7f207819bc90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2074009fb0 tx=0x7f2074011a20 comp rx=0 tx=0).stop 2024-03-11T20:46:39.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.635+0000 7f207f45d700 1 -- 172.21.15.129:0/1713423158 shutdown_connections 2024-03-11T20:46:39.190 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.635+0000 7f207f45d700 1 --2- 172.21.15.129:0/1713423158 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2064038fc0 0x7f206403b480 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:39.191 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.635+0000 7f207f45d700 1 --2- 172.21.15.129:0/1713423158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2078105c60 0x7f207819bc90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:39.191 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.635+0000 7f207f45d700 1 -- 172.21.15.129:0/1713423158 >> 172.21.15.129:0/1713423158 conn(0x7f20781011c0 msgr2=0x7f2078101cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:39.191 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.636+0000 7f207f45d700 1 -- 172.21.15.129:0/1713423158 shutdown_connections 2024-03-11T20:46:39.191 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr 2024-03-11T20:46:38.636+0000 7f207f45d700 1 -- 172.21.15.129:0/1713423158 wait complete. 2024-03-11T20:46:39.191 INFO:teuthology.orchestra.run.smithi129.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-11T20:46:39.191 INFO:teuthology.orchestra.run.smithi129.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-11T20:46:39.191 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:46:39.191 INFO:teuthology.orchestra.run.smithi129.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T20:46:39.191 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:46:39.191 INFO:teuthology.orchestra.run.smithi129.stdout:Or, if you are only running a single cluster on this host: 2024-03-11T20:46:39.191 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:46:39.191 INFO:teuthology.orchestra.run.smithi129.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-11T20:46:39.191 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:46:39.191 INFO:teuthology.orchestra.run.smithi129.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-11T20:46:39.192 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:46:39.192 INFO:teuthology.orchestra.run.smithi129.stdout: ceph telemetry on 2024-03-11T20:46:39.192 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:46:39.192 INFO:teuthology.orchestra.run.smithi129.stdout:For more information see: 2024-03-11T20:46:39.192 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:46:39.192 INFO:teuthology.orchestra.run.smithi129.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-11T20:46:39.192 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:46:39.192 INFO:teuthology.orchestra.run.smithi129.stdout:Bootstrap complete. 2024-03-11T20:46:39.227 INFO:tasks.cephadm:Fetching config... 2024-03-11T20:46:39.228 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:46:39.228 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T20:46:39.243 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T20:46:39.243 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:46:39.243 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T20:46:39.299 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T20:46:39.299 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:46:39.299 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-11T20:46:39.366 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T20:46:39.366 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:46:39.366 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T20:46:39.421 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T20:46:39.421 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2skFy55BXJHmBwHv2vClbQr5gHVeXZUepErQ2bx63MX+w3TwLc5KHSWM+Zr67WfhAeH/ToOJaeP6fY7yvTn5HWv4vEXtoqreW4wz0XD4JCCcjFTlR25DCADV68prbishdZOnQEwnQSOPFN6wj6DAtOqtlcZGLYz7sI/QDRejQUEGyaCEb1ZNTH3P7KBYLFUHnZ6pO63PietHinDhyaq/kCI58N3ND9rEiIyeqfjHG8q8bIapTyVMig/AmAIFjRHmBhlzdl+Y6cAKAmLa3r6ek5yp19ufDv8u378Kmki1u1M1N3YYjbKbGvZpwaNEkKkWNB2zalfOyNetDXwxYmaiK6LPhNzVvDRKB1a6sxAApNfUZibiLdmDkvsgh5YtI56HOgXVRSjTJgeexB2+7FXiBmFhx705HJ8APPNS6YkYyXfBRqQIyjVZhfrsdo+qWH7kl/99EEAI53Fcwo4KVhm89PHwAfjsgtLl1WNyoJSgCJThSV7LGpxwn0FL6ytnIqUc= ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T20:46:39.500 INFO:teuthology.orchestra.run.smithi129.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2skFy55BXJHmBwHv2vClbQr5gHVeXZUepErQ2bx63MX+w3TwLc5KHSWM+Zr67WfhAeH/ToOJaeP6fY7yvTn5HWv4vEXtoqreW4wz0XD4JCCcjFTlR25DCADV68prbishdZOnQEwnQSOPFN6wj6DAtOqtlcZGLYz7sI/QDRejQUEGyaCEb1ZNTH3P7KBYLFUHnZ6pO63PietHinDhyaq/kCI58N3ND9rEiIyeqfjHG8q8bIapTyVMig/AmAIFjRHmBhlzdl+Y6cAKAmLa3r6ek5yp19ufDv8u378Kmki1u1M1N3YYjbKbGvZpwaNEkKkWNB2zalfOyNetDXwxYmaiK6LPhNzVvDRKB1a6sxAApNfUZibiLdmDkvsgh5YtI56HOgXVRSjTJgeexB2+7FXiBmFhx705HJ8APPNS6YkYyXfBRqQIyjVZhfrsdo+qWH7kl/99EEAI53Fcwo4KVhm89PHwAfjsgtLl1WNyoJSgCJThSV7LGpxwn0FL6ytnIqUc= ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:46:39.514 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2skFy55BXJHmBwHv2vClbQr5gHVeXZUepErQ2bx63MX+w3TwLc5KHSWM+Zr67WfhAeH/ToOJaeP6fY7yvTn5HWv4vEXtoqreW4wz0XD4JCCcjFTlR25DCADV68prbishdZOnQEwnQSOPFN6wj6DAtOqtlcZGLYz7sI/QDRejQUEGyaCEb1ZNTH3P7KBYLFUHnZ6pO63PietHinDhyaq/kCI58N3ND9rEiIyeqfjHG8q8bIapTyVMig/AmAIFjRHmBhlzdl+Y6cAKAmLa3r6ek5yp19ufDv8u378Kmki1u1M1N3YYjbKbGvZpwaNEkKkWNB2zalfOyNetDXwxYmaiK6LPhNzVvDRKB1a6sxAApNfUZibiLdmDkvsgh5YtI56HOgXVRSjTJgeexB2+7FXiBmFhx705HJ8APPNS6YkYyXfBRqQIyjVZhfrsdo+qWH7kl/99EEAI53Fcwo4KVhm89PHwAfjsgtLl1WNyoJSgCJThSV7LGpxwn0FL6ytnIqUc= ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T20:46:39.560 INFO:teuthology.orchestra.run.smithi157.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2skFy55BXJHmBwHv2vClbQr5gHVeXZUepErQ2bx63MX+w3TwLc5KHSWM+Zr67WfhAeH/ToOJaeP6fY7yvTn5HWv4vEXtoqreW4wz0XD4JCCcjFTlR25DCADV68prbishdZOnQEwnQSOPFN6wj6DAtOqtlcZGLYz7sI/QDRejQUEGyaCEb1ZNTH3P7KBYLFUHnZ6pO63PietHinDhyaq/kCI58N3ND9rEiIyeqfjHG8q8bIapTyVMig/AmAIFjRHmBhlzdl+Y6cAKAmLa3r6ek5yp19ufDv8u378Kmki1u1M1N3YYjbKbGvZpwaNEkKkWNB2zalfOyNetDXwxYmaiK6LPhNzVvDRKB1a6sxAApNfUZibiLdmDkvsgh5YtI56HOgXVRSjTJgeexB2+7FXiBmFhx705HJ8APPNS6YkYyXfBRqQIyjVZhfrsdo+qWH7kl/99EEAI53Fcwo4KVhm89PHwAfjsgtLl1WNyoJSgCJThSV7LGpxwn0FL6ytnIqUc= ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:46:39.575 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T20:46:39.782 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1713423158' entity='client.admin' 2024-03-11T20:46:39.852 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:46:42.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.527+0000 7fb1ed8bb700 1 -- 172.21.15.129:0/112663330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1e80ffb50 msgr2=0x7fb1e80fff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:42.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.527+0000 7fb1ed8bb700 1 --2- 172.21.15.129:0/112663330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1e80ffb50 0x7fb1e80fff70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8009910 tx=0x7fb1d8009c20 comp rx=0 tx=0).stop 2024-03-11T20:46:42.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.527+0000 7fb1ed8bb700 1 -- 172.21.15.129:0/112663330 shutdown_connections 2024-03-11T20:46:42.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.527+0000 7fb1ed8bb700 1 --2- 172.21.15.129:0/112663330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1e80ffb50 0x7fb1e80fff70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.527+0000 7fb1ed8bb700 1 -- 172.21.15.129:0/112663330 >> 172.21.15.129:0/112663330 conn(0x7fb1e80fb490 msgr2=0x7fb1e80fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.527+0000 7fb1ed8bb700 1 -- 172.21.15.129:0/112663330 shutdown_connections 2024-03-11T20:46:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.527+0000 7fb1ed8bb700 1 -- 172.21.15.129:0/112663330 wait complete. 2024-03-11T20:46:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.527+0000 7fb1ed8bb700 1 Processor -- start 2024-03-11T20:46:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.528+0000 7fb1ed8bb700 1 -- start start 2024-03-11T20:46:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.528+0000 7fb1ed8bb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1e80ffb50 0x7fb1e8102450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.528+0000 7fb1ed8bb700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e8106cb0 con 0x7fb1e80ffb50 2024-03-11T20:46:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.528+0000 7fb1e6ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1e80ffb50 0x7fb1e8102450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.528+0000 7fb1e6ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1e80ffb50 0x7fb1e8102450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47642/0 (socket says 172.21.15.129:47642) 2024-03-11T20:46:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.528+0000 7fb1e6ffd700 1 -- 172.21.15.129:0/4048333409 learned_addr learned my addr 172.21.15.129:0/4048333409 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.528+0000 7fb1e6ffd700 1 -- 172.21.15.129:0/4048333409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e8102990 con 0x7fb1e80ffb50 2024-03-11T20:46:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.528+0000 7fb1e6ffd700 1 --2- 172.21.15.129:0/4048333409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1e80ffb50 0x7fb1e8102450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8005950 tx=0x7fb1d80057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.528+0000 7fb1ec8b9700 1 -- 172.21.15.129:0/4048333409 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1d8027070 con 0x7fb1e80ffb50 2024-03-11T20:46:42.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.528+0000 7fb1ec8b9700 1 -- 172.21.15.129:0/4048333409 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb1d800f460 con 0x7fb1e80ffb50 2024-03-11T20:46:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.529+0000 7fb1ed8bb700 1 -- 172.21.15.129:0/4048333409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e8100b80 con 0x7fb1e80ffb50 2024-03-11T20:46:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.529+0000 7fb1ec8b9700 1 -- 172.21.15.129:0/4048333409 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1d8017440 con 0x7fb1e80ffb50 2024-03-11T20:46:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.529+0000 7fb1ed8bb700 1 -- 172.21.15.129:0/4048333409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e81010c0 con 0x7fb1e80ffb50 2024-03-11T20:46:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.529+0000 7fb1ec8b9700 1 -- 172.21.15.129:0/4048333409 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fb1d802a070 con 0x7fb1e80ffb50 2024-03-11T20:46:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.530+0000 7fb1ec8b9700 1 --2- 172.21.15.129:0/4048333409 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb1d0038fc0 0x7fb1d003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.530+0000 7fb1ec8b9700 1 -- 172.21.15.129:0/4048333409 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb1d804d100 con 0x7fb1e80ffb50 2024-03-11T20:46:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.530+0000 7fb1e67fc700 1 --2- 172.21.15.129:0/4048333409 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb1d0038fc0 0x7fb1d003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.530+0000 7fb1ed8bb700 1 -- 172.21.15.129:0/4048333409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e8107430 con 0x7fb1e80ffb50 2024-03-11T20:46:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.530+0000 7fb1e67fc700 1 --2- 172.21.15.129:0/4048333409 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb1d0038fc0 0x7fb1d003b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc006fd0 tx=0x7fb1dc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:42.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.535+0000 7fb1ec8b9700 1 -- 172.21.15.129:0/4048333409 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fb1d8031070 con 0x7fb1e80ffb50 2024-03-11T20:46:42.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.738+0000 7fb1ed8bb700 1 -- 172.21.15.129:0/4048333409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fb1e818fa90 con 0x7fb1e80ffb50 2024-03-11T20:46:42.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.740+0000 7fb1ec8b9700 1 -- 172.21.15.129:0/4048333409 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fb1d8013070 con 0x7fb1e80ffb50 2024-03-11T20:46:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.743+0000 7fb1ed8bb700 1 -- 172.21.15.129:0/4048333409 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb1d0038fc0 msgr2=0x7fb1d003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.743+0000 7fb1ed8bb700 1 --2- 172.21.15.129:0/4048333409 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb1d0038fc0 0x7fb1d003b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc006fd0 tx=0x7fb1dc006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.743+0000 7fb1ed8bb700 1 -- 172.21.15.129:0/4048333409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1e80ffb50 msgr2=0x7fb1e8102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.743+0000 7fb1ed8bb700 1 --2- 172.21.15.129:0/4048333409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1e80ffb50 0x7fb1e8102450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8005950 tx=0x7fb1d80057c0 comp rx=0 tx=0).stop 2024-03-11T20:46:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.743+0000 7fb1ed8bb700 1 -- 172.21.15.129:0/4048333409 shutdown_connections 2024-03-11T20:46:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.743+0000 7fb1ed8bb700 1 --2- 172.21.15.129:0/4048333409 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb1d0038fc0 0x7fb1d003b480 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.743+0000 7fb1ed8bb700 1 --2- 172.21.15.129:0/4048333409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1e80ffb50 0x7fb1e8102450 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.743+0000 7fb1ed8bb700 1 -- 172.21.15.129:0/4048333409 >> 172.21.15.129:0/4048333409 conn(0x7fb1e80fb490 msgr2=0x7fb1e80fbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.743+0000 7fb1ed8bb700 1 -- 172.21.15.129:0/4048333409 shutdown_connections 2024-03-11T20:46:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:42.743+0000 7fb1ed8bb700 1 -- 172.21.15.129:0/4048333409 wait complete. 2024-03-11T20:46:43.345 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T20:46:43.346 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T20:46:43.917 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:46:44.039 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4048333409' entity='client.admin' 2024-03-11T20:46:44.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:44 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:44.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:44 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:44.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:44 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:46:44.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:44 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:44.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:44 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:44.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:44 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:45.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.476+0000 7f9b95127700 1 -- 172.21.15.129:0/2019049230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b90101830 msgr2=0x7f9b90101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:45.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.476+0000 7f9b95127700 1 --2- 172.21.15.129:0/2019049230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b90101830 0x7f9b90101c50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009910 tx=0x7f9b80009c20 comp rx=0 tx=0).stop 2024-03-11T20:46:45.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.476+0000 7f9b95127700 1 -- 172.21.15.129:0/2019049230 shutdown_connections 2024-03-11T20:46:45.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.476+0000 7f9b95127700 1 --2- 172.21.15.129:0/2019049230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b90101830 0x7f9b90101c50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:45.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.476+0000 7f9b95127700 1 -- 172.21.15.129:0/2019049230 >> 172.21.15.129:0/2019049230 conn(0x7f9b900fcdd0 msgr2=0x7f9b900ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:45.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.476+0000 7f9b95127700 1 -- 172.21.15.129:0/2019049230 shutdown_connections 2024-03-11T20:46:45.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.476+0000 7f9b95127700 1 -- 172.21.15.129:0/2019049230 wait complete. 2024-03-11T20:46:45.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.477+0000 7f9b95127700 1 Processor -- start 2024-03-11T20:46:45.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.477+0000 7f9b95127700 1 -- start start 2024-03-11T20:46:45.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.477+0000 7f9b95127700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b90101830 0x7f9b901964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:45.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.477+0000 7f9b95127700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b90106d30 con 0x7f9b90101830 2024-03-11T20:46:45.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.478+0000 7f9b8ed9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b90101830 0x7f9b901964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:45.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.478+0000 7f9b8ed9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b90101830 0x7f9b901964a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37756/0 (socket says 172.21.15.129:37756) 2024-03-11T20:46:45.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.478+0000 7f9b8ed9d700 1 -- 172.21.15.129:0/919449033 learned_addr learned my addr 172.21.15.129:0/919449033 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:45.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.478+0000 7f9b8ed9d700 1 -- 172.21.15.129:0/919449033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b901969e0 con 0x7f9b90101830 2024-03-11T20:46:45.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.478+0000 7f9b8ed9d700 1 --2- 172.21.15.129:0/919449033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b90101830 0x7f9b901964a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9b80005950 tx=0x7f9b800057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:45.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.478+0000 7f9b77fff700 1 -- 172.21.15.129:0/919449033 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b80027070 con 0x7f9b90101830 2024-03-11T20:46:45.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.478+0000 7f9b95127700 1 -- 172.21.15.129:0/919449033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b90196c70 con 0x7f9b90101830 2024-03-11T20:46:45.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.478+0000 7f9b77fff700 1 -- 172.21.15.129:0/919449033 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9b8000f460 con 0x7f9b90101830 2024-03-11T20:46:45.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.478+0000 7f9b77fff700 1 -- 172.21.15.129:0/919449033 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b80017440 con 0x7f9b90101830 2024-03-11T20:46:45.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.478+0000 7f9b95127700 1 -- 172.21.15.129:0/919449033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b90199960 con 0x7f9b90101830 2024-03-11T20:46:45.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.479+0000 7f9b77fff700 1 -- 172.21.15.129:0/919449033 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f9b8002a070 con 0x7f9b90101830 2024-03-11T20:46:45.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.479+0000 7f9b77fff700 1 --2- 172.21.15.129:0/919449033 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f9b78038fc0 0x7f9b7803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:45.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.479+0000 7f9b77fff700 1 -- 172.21.15.129:0/919449033 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9b8004d110 con 0x7f9b90101830 2024-03-11T20:46:45.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.479+0000 7f9b95127700 1 -- 172.21.15.129:0/919449033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b901074b0 con 0x7f9b90101830 2024-03-11T20:46:45.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.479+0000 7f9b8e59c700 1 --2- 172.21.15.129:0/919449033 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f9b78038fc0 0x7f9b7803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:45.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.480+0000 7f9b8e59c700 1 --2- 172.21.15.129:0/919449033 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f9b78038fc0 0x7f9b7803b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9b84006fd0 tx=0x7f9b84006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:45.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.484+0000 7f9b77fff700 1 -- 172.21.15.129:0/919449033 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f9b80031070 con 0x7f9b90101830 2024-03-11T20:46:45.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.681+0000 7f9b95127700 1 -- 172.21.15.129:0/919449033 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b90062220 con 0x7f9b78038fc0 2024-03-11T20:46:45.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.684+0000 7f9b77fff700 1 -- 172.21.15.129:0/919449033 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9b90062220 con 0x7f9b78038fc0 2024-03-11T20:46:45.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.686+0000 7f9b95127700 1 -- 172.21.15.129:0/919449033 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f9b78038fc0 msgr2=0x7f9b7803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:45.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.686+0000 7f9b95127700 1 --2- 172.21.15.129:0/919449033 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f9b78038fc0 0x7f9b7803b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9b84006fd0 tx=0x7f9b84006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:45.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.686+0000 7f9b95127700 1 -- 172.21.15.129:0/919449033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b90101830 msgr2=0x7f9b901964a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:45.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.686+0000 7f9b95127700 1 --2- 172.21.15.129:0/919449033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b90101830 0x7f9b901964a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9b80005950 tx=0x7f9b800057c0 comp rx=0 tx=0).stop 2024-03-11T20:46:45.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.686+0000 7f9b95127700 1 -- 172.21.15.129:0/919449033 shutdown_connections 2024-03-11T20:46:45.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.686+0000 7f9b95127700 1 --2- 172.21.15.129:0/919449033 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f9b78038fc0 0x7f9b7803b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:45.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.686+0000 7f9b95127700 1 --2- 172.21.15.129:0/919449033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b90101830 0x7f9b901964a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:45.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.686+0000 7f9b95127700 1 -- 172.21.15.129:0/919449033 >> 172.21.15.129:0/919449033 conn(0x7f9b900fcdd0 msgr2=0x7f9b900fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:45.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.687+0000 7f9b95127700 1 -- 172.21.15.129:0/919449033 shutdown_connections 2024-03-11T20:46:45.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:45.687+0000 7f9b95127700 1 -- 172.21.15.129:0/919449033 wait complete. 2024-03-11T20:46:46.452 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi157 2024-03-11T20:46:46.453 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-11T20:46:46.453 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/etc/ceph/ceph.conf 2024-03-11T20:46:46.472 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-11T20:46:46.472 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:46:46.530 INFO:tasks.cephadm:Adding host smithi157 to orchestrator... 2024-03-11T20:46:46.531 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch host add smithi157 2024-03-11T20:46:46.763 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:46:46.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:46 smithi129 ceph-mon[30184]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:46:46.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:46.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:46.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:46.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:46:46.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:46 smithi129 ceph-mon[30184]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-11T20:46:46.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:46.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:46.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:47.935 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:47 smithi129 ceph-mon[30184]: Updating smithi129:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:46:47.935 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:47 smithi129 ceph-mon[30184]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:46:47.935 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:47 smithi129 ceph-mon[30184]: Updating smithi129:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:46:48.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.288+0000 7f2f8159c700 1 -- 172.21.15.129:0/2647119824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c106690 msgr2=0x7f2f7c106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:48.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.288+0000 7f2f8159c700 1 --2- 172.21.15.129:0/2647119824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c106690 0x7f2f7c106a70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009910 tx=0x7f2f6c009c20 comp rx=0 tx=0).stop 2024-03-11T20:46:48.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.288+0000 7f2f8159c700 1 -- 172.21.15.129:0/2647119824 shutdown_connections 2024-03-11T20:46:48.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.288+0000 7f2f8159c700 1 --2- 172.21.15.129:0/2647119824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c106690 0x7f2f7c106a70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:48.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.288+0000 7f2f8159c700 1 -- 172.21.15.129:0/2647119824 >> 172.21.15.129:0/2647119824 conn(0x7f2f7c077480 msgr2=0x7f2f7c0fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:48.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.288+0000 7f2f8159c700 1 -- 172.21.15.129:0/2647119824 shutdown_connections 2024-03-11T20:46:48.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.288+0000 7f2f8159c700 1 -- 172.21.15.129:0/2647119824 wait complete. 2024-03-11T20:46:48.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.289+0000 7f2f8159c700 1 Processor -- start 2024-03-11T20:46:48.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.289+0000 7f2f8159c700 1 -- start start 2024-03-11T20:46:48.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.289+0000 7f2f8159c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c106690 0x7f2f7c1954f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:48.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.289+0000 7f2f8159c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7c10a150 con 0x7f2f7c106690 2024-03-11T20:46:48.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.289+0000 7f2f7affd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c106690 0x7f2f7c1954f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:48.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.289+0000 7f2f7affd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c106690 0x7f2f7c1954f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37780/0 (socket says 172.21.15.129:37780) 2024-03-11T20:46:48.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.289+0000 7f2f7affd700 1 -- 172.21.15.129:0/457719812 learned_addr learned my addr 172.21.15.129:0/457719812 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:48.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.290+0000 7f2f7affd700 1 -- 172.21.15.129:0/457719812 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c069fe0 con 0x7f2f7c106690 2024-03-11T20:46:48.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.290+0000 7f2f7affd700 1 --2- 172.21.15.129:0/457719812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c106690 0x7f2f7c1954f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c006010 tx=0x7f2f6c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:48.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.290+0000 7f2f63fff700 1 -- 172.21.15.129:0/457719812 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f6c027070 con 0x7f2f7c106690 2024-03-11T20:46:48.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.290+0000 7f2f63fff700 1 -- 172.21.15.129:0/457719812 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2f6c022470 con 0x7f2f7c106690 2024-03-11T20:46:48.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.290+0000 7f2f8159c700 1 -- 172.21.15.129:0/457719812 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7c195a30 con 0x7f2f7c106690 2024-03-11T20:46:48.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.290+0000 7f2f63fff700 1 -- 172.21.15.129:0/457719812 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f6c017440 con 0x7f2f7c106690 2024-03-11T20:46:48.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.290+0000 7f2f8159c700 1 -- 172.21.15.129:0/457719812 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7c195ef0 con 0x7f2f7c106690 2024-03-11T20:46:48.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.291+0000 7f2f63fff700 1 -- 172.21.15.129:0/457719812 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f2f6c02a070 con 0x7f2f7c106690 2024-03-11T20:46:48.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.291+0000 7f2f63fff700 1 --2- 172.21.15.129:0/457719812 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2f64039020 0x7f2f6403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:48.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.291+0000 7f2f63fff700 1 -- 172.21.15.129:0/457719812 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2f6c01e030 con 0x7f2f7c106690 2024-03-11T20:46:48.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.291+0000 7f2f7a7fc700 1 --2- 172.21.15.129:0/457719812 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2f64039020 0x7f2f6403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:48.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.291+0000 7f2f8159c700 1 -- 172.21.15.129:0/457719812 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f7c101dd0 con 0x7f2f7c106690 2024-03-11T20:46:48.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.291+0000 7f2f7a7fc700 1 --2- 172.21.15.129:0/457719812 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2f64039020 0x7f2f6403b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2f70006fd0 tx=0x7f2f70006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:48.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.296+0000 7f2f63fff700 1 -- 172.21.15.129:0/457719812 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f2f6c013070 con 0x7f2f7c106690 2024-03-11T20:46:48.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:48.493+0000 7f2f8159c700 1 -- 172.21.15.129:0/457719812 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi157", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f7c063410 con 0x7f2f64039020 2024-03-11T20:46:49.227 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:48 smithi129 ceph-mon[30184]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi157", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:46:50.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:50 smithi129 ceph-mon[30184]: Deploying cephadm binary to smithi157 2024-03-11T20:46:50.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:50 smithi129 ceph-mon[30184]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:46:52.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:52.828+0000 7f2f63fff700 1 -- 172.21.15.129:0/457719812 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f2f7c063410 con 0x7f2f64039020 2024-03-11T20:46:52.824 INFO:teuthology.orchestra.run.smithi129.stdout:Added host 'smithi157' with addr '172.21.15.157' 2024-03-11T20:46:52.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:52.831+0000 7f2f8159c700 1 -- 172.21.15.129:0/457719812 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2f64039020 msgr2=0x7f2f6403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:52.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:52.831+0000 7f2f8159c700 1 --2- 172.21.15.129:0/457719812 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2f64039020 0x7f2f6403b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2f70006fd0 tx=0x7f2f70006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:52.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:52.831+0000 7f2f8159c700 1 -- 172.21.15.129:0/457719812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c106690 msgr2=0x7f2f7c1954f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:52.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:52.831+0000 7f2f8159c700 1 --2- 172.21.15.129:0/457719812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c106690 0x7f2f7c1954f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c006010 tx=0x7f2f6c00f740 comp rx=0 tx=0).stop 2024-03-11T20:46:52.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:52.831+0000 7f2f8159c700 1 -- 172.21.15.129:0/457719812 shutdown_connections 2024-03-11T20:46:52.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:52.831+0000 7f2f8159c700 1 --2- 172.21.15.129:0/457719812 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2f64039020 0x7f2f6403b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:52.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:52.831+0000 7f2f8159c700 1 --2- 172.21.15.129:0/457719812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c106690 0x7f2f7c1954f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:52.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:52.831+0000 7f2f8159c700 1 -- 172.21.15.129:0/457719812 >> 172.21.15.129:0/457719812 conn(0x7f2f7c077480 msgr2=0x7f2f7c104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:52.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:52.831+0000 7f2f8159c700 1 -- 172.21.15.129:0/457719812 shutdown_connections 2024-03-11T20:46:52.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:52.831+0000 7f2f8159c700 1 -- 172.21.15.129:0/457719812 wait complete. 2024-03-11T20:46:52.837 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:52 smithi129 ceph-mon[30184]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:46:53.483 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T20:46:53.720 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:46:54.084 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:53 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:54.084 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:53 smithi129 ceph-mon[30184]: Added host smithi157 2024-03-11T20:46:54.084 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:53 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:46:54.085 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:53 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:54.085 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:53 smithi129 ceph-mon[30184]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:46:55.106 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:55.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.346+0000 7fb2cee0a700 1 -- 172.21.15.129:0/3702477493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c8100770 msgr2=0x7fb2c8100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:55.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.346+0000 7fb2cee0a700 1 --2- 172.21.15.129:0/3702477493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c8100770 0x7fb2c8100b50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc009910 tx=0x7fb2bc009c20 comp rx=0 tx=0).stop 2024-03-11T20:46:55.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.346+0000 7fb2cee0a700 1 -- 172.21.15.129:0/3702477493 shutdown_connections 2024-03-11T20:46:55.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.346+0000 7fb2cee0a700 1 --2- 172.21.15.129:0/3702477493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c8100770 0x7fb2c8100b50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:55.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.346+0000 7fb2cee0a700 1 -- 172.21.15.129:0/3702477493 >> 172.21.15.129:0/3702477493 conn(0x7fb2c80fc1f0 msgr2=0x7fb2c80fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:55.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.346+0000 7fb2cee0a700 1 -- 172.21.15.129:0/3702477493 shutdown_connections 2024-03-11T20:46:55.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.346+0000 7fb2cee0a700 1 -- 172.21.15.129:0/3702477493 wait complete. 2024-03-11T20:46:55.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.347+0000 7fb2cee0a700 1 Processor -- start 2024-03-11T20:46:55.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.347+0000 7fb2cee0a700 1 -- start start 2024-03-11T20:46:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.347+0000 7fb2cee0a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c8100770 0x7fb2c819cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.347+0000 7fb2cee0a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c81118d0 con 0x7fb2c8100770 2024-03-11T20:46:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.347+0000 7fb2ccba6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c8100770 0x7fb2c819cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.347+0000 7fb2ccba6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c8100770 0x7fb2c819cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37806/0 (socket says 172.21.15.129:37806) 2024-03-11T20:46:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.347+0000 7fb2ccba6700 1 -- 172.21.15.129:0/3616457067 learned_addr learned my addr 172.21.15.129:0/3616457067 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.348+0000 7fb2ccba6700 1 -- 172.21.15.129:0/3616457067 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c819d450 con 0x7fb2c8100770 2024-03-11T20:46:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.348+0000 7fb2ccba6700 1 --2- 172.21.15.129:0/3616457067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c8100770 0x7fb2c819cf10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc006010 tx=0x7fb2bc00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.348+0000 7fb2c5ffb700 1 -- 172.21.15.129:0/3616457067 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2bc027070 con 0x7fb2c8100770 2024-03-11T20:46:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.348+0000 7fb2cee0a700 1 -- 172.21.15.129:0/3616457067 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c819d6e0 con 0x7fb2c8100770 2024-03-11T20:46:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.348+0000 7fb2c5ffb700 1 -- 172.21.15.129:0/3616457067 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb2bc022470 con 0x7fb2c8100770 2024-03-11T20:46:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.348+0000 7fb2c5ffb700 1 -- 172.21.15.129:0/3616457067 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2bc017440 con 0x7fb2c8100770 2024-03-11T20:46:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.348+0000 7fb2cee0a700 1 -- 172.21.15.129:0/3616457067 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c819db40 con 0x7fb2c8100770 2024-03-11T20:46:55.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.349+0000 7fb2c5ffb700 1 -- 172.21.15.129:0/3616457067 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fb2bc02a070 con 0x7fb2c8100770 2024-03-11T20:46:55.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.349+0000 7fb2cee0a700 1 -- 172.21.15.129:0/3616457067 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c806a4b0 con 0x7fb2c8100770 2024-03-11T20:46:55.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.349+0000 7fb2c5ffb700 1 --2- 172.21.15.129:0/3616457067 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb2b8038fc0 0x7fb2b803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:55.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.349+0000 7fb2c5ffb700 1 -- 172.21.15.129:0/3616457067 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb2bc01e030 con 0x7fb2c8100770 2024-03-11T20:46:55.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.349+0000 7fb2c7fff700 1 --2- 172.21.15.129:0/3616457067 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb2b8038fc0 0x7fb2b803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:55.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.350+0000 7fb2c7fff700 1 --2- 172.21.15.129:0/3616457067 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb2b8038fc0 0x7fb2b803b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4006fd0 tx=0x7fb2b4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:55.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.354+0000 7fb2c5ffb700 1 -- 172.21.15.129:0/3616457067 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fb2bc013070 con 0x7fb2c8100770 2024-03-11T20:46:55.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.553+0000 7fb2cee0a700 1 -- 172.21.15.129:0/3616457067 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2c819e220 con 0x7fb2b8038fc0 2024-03-11T20:46:55.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.554+0000 7fb2c5ffb700 1 -- 172.21.15.129:0/3616457067 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7fb2c819e220 con 0x7fb2b8038fc0 2024-03-11T20:46:55.549 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:46:55.549 INFO:teuthology.orchestra.run.smithi129.stdout:[{"addr": "172.21.15.129", "hostname": "smithi129", "labels": [], "status": ""}, {"addr": "172.21.15.157", "hostname": "smithi157", "labels": [], "status": ""}] 2024-03-11T20:46:55.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.556+0000 7fb2cee0a700 1 -- 172.21.15.129:0/3616457067 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb2b8038fc0 msgr2=0x7fb2b803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:55.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.556+0000 7fb2cee0a700 1 --2- 172.21.15.129:0/3616457067 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb2b8038fc0 0x7fb2b803b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4006fd0 tx=0x7fb2b4006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:55.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.556+0000 7fb2cee0a700 1 -- 172.21.15.129:0/3616457067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c8100770 msgr2=0x7fb2c819cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:55.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.556+0000 7fb2cee0a700 1 --2- 172.21.15.129:0/3616457067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c8100770 0x7fb2c819cf10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc006010 tx=0x7fb2bc00f740 comp rx=0 tx=0).stop 2024-03-11T20:46:55.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.557+0000 7fb2cee0a700 1 -- 172.21.15.129:0/3616457067 shutdown_connections 2024-03-11T20:46:55.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.557+0000 7fb2cee0a700 1 --2- 172.21.15.129:0/3616457067 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb2b8038fc0 0x7fb2b803b480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:55.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.557+0000 7fb2cee0a700 1 --2- 172.21.15.129:0/3616457067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c8100770 0x7fb2c819cf10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:55.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.557+0000 7fb2cee0a700 1 -- 172.21.15.129:0/3616457067 >> 172.21.15.129:0/3616457067 conn(0x7fb2c80fc1f0 msgr2=0x7fb2c81101d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:55.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.557+0000 7fb2cee0a700 1 -- 172.21.15.129:0/3616457067 shutdown_connections 2024-03-11T20:46:55.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:55.557+0000 7fb2cee0a700 1 -- 172.21.15.129:0/3616457067 wait complete. 2024-03-11T20:46:56.140 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:55 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:56.140 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:55 smithi129 ceph-mon[30184]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:46:56.140 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:55 smithi129 ceph-mon[30184]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:46:56.169 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T20:46:56.170 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T20:46:56.400 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:46:57.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.928+0000 7f4979116700 1 -- 172.21.15.129:0/31467909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49741077a0 msgr2=0x7f4974107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:57.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.928+0000 7f4979116700 1 --2- 172.21.15.129:0/31467909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49741077a0 0x7f4974107b80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4964009910 tx=0x7f4964009c20 comp rx=0 tx=0).stop 2024-03-11T20:46:57.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.928+0000 7f4979116700 1 -- 172.21.15.129:0/31467909 shutdown_connections 2024-03-11T20:46:57.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.928+0000 7f4979116700 1 --2- 172.21.15.129:0/31467909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49741077a0 0x7f4974107b80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:57.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.928+0000 7f4979116700 1 -- 172.21.15.129:0/31467909 >> 172.21.15.129:0/31467909 conn(0x7f49740fd2b0 msgr2=0x7f49740ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:57.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.928+0000 7f4979116700 1 -- 172.21.15.129:0/31467909 shutdown_connections 2024-03-11T20:46:57.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.928+0000 7f4979116700 1 -- 172.21.15.129:0/31467909 wait complete. 2024-03-11T20:46:57.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.928+0000 7f4979116700 1 Processor -- start 2024-03-11T20:46:57.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.929+0000 7f4979116700 1 -- start start 2024-03-11T20:46:57.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.929+0000 7f4979116700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49741077a0 0x7f49740740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:57.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.929+0000 7f4979116700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497410b260 con 0x7f49741077a0 2024-03-11T20:46:57.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.929+0000 7f4972d9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49741077a0 0x7f49740740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:57.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.929+0000 7f4972d9d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49741077a0 0x7f49740740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38662/0 (socket says 172.21.15.129:38662) 2024-03-11T20:46:57.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.929+0000 7f4972d9d700 1 -- 172.21.15.129:0/2115522764 learned_addr learned my addr 172.21.15.129:0/2115522764 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:46:57.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.929+0000 7f4972d9d700 1 -- 172.21.15.129:0/2115522764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4974075fd0 con 0x7f49741077a0 2024-03-11T20:46:57.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.929+0000 7f4972d9d700 1 --2- 172.21.15.129:0/2115522764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49741077a0 0x7f49740740e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4964006010 tx=0x7f496400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:57.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.929+0000 7f4970d99700 1 -- 172.21.15.129:0/2115522764 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4964027070 con 0x7f49741077a0 2024-03-11T20:46:57.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.929+0000 7f4970d99700 1 -- 172.21.15.129:0/2115522764 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4964022470 con 0x7f49741077a0 2024-03-11T20:46:57.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.929+0000 7f4979116700 1 -- 172.21.15.129:0/2115522764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4974074680 con 0x7f49741077a0 2024-03-11T20:46:57.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.929+0000 7f4970d99700 1 -- 172.21.15.129:0/2115522764 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4964017440 con 0x7f49741077a0 2024-03-11T20:46:57.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.929+0000 7f4979116700 1 -- 172.21.15.129:0/2115522764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4974074b40 con 0x7f49741077a0 2024-03-11T20:46:57.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.930+0000 7f4970d99700 1 -- 172.21.15.129:0/2115522764 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f496402a070 con 0x7f49741077a0 2024-03-11T20:46:57.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.930+0000 7f4979116700 1 -- 172.21.15.129:0/2115522764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4974102ee0 con 0x7f49741077a0 2024-03-11T20:46:57.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.930+0000 7f4970d99700 1 --2- 172.21.15.129:0/2115522764 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4960039020 0x7f496003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:46:57.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.930+0000 7f4970d99700 1 -- 172.21.15.129:0/2115522764 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f496401e030 con 0x7f49741077a0 2024-03-11T20:46:57.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.931+0000 7f496bfff700 1 --2- 172.21.15.129:0/2115522764 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4960039020 0x7f496003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:46:57.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.931+0000 7f496bfff700 1 --2- 172.21.15.129:0/2115522764 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4960039020 0x7f496003b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f495c006fd0 tx=0x7f495c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:46:57.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:57.935+0000 7f4970d99700 1 -- 172.21.15.129:0/2115522764 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f4964013070 con 0x7f49741077a0 2024-03-11T20:46:58.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:58.156+0000 7f4979116700 1 -- 172.21.15.129:0/2115522764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f4974063410 con 0x7f49741077a0 2024-03-11T20:46:58.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:58 smithi129 ceph-mon[30184]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:46:58.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:58 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:58.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:58 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:58.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:58 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:58.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:58 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:58.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:58 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi157", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:46:58.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:58 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:46:58.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:58 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:46:58.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2115522764' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T20:46:59.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:59.063+0000 7f4970d99700 1 -- 172.21.15.129:0/2115522764 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4964029d80 con 0x7f49741077a0 2024-03-11T20:46:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:59.065+0000 7f4979116700 1 -- 172.21.15.129:0/2115522764 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4960039020 msgr2=0x7f496003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:59.065+0000 7f4979116700 1 --2- 172.21.15.129:0/2115522764 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4960039020 0x7f496003b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f495c006fd0 tx=0x7f495c006e40 comp rx=0 tx=0).stop 2024-03-11T20:46:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:59.065+0000 7f4979116700 1 -- 172.21.15.129:0/2115522764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49741077a0 msgr2=0x7f49740740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:46:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:59.065+0000 7f4979116700 1 --2- 172.21.15.129:0/2115522764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49741077a0 0x7f49740740e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4964006010 tx=0x7f496400f740 comp rx=0 tx=0).stop 2024-03-11T20:46:59.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:59.065+0000 7f4979116700 1 -- 172.21.15.129:0/2115522764 shutdown_connections 2024-03-11T20:46:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:59.065+0000 7f4979116700 1 --2- 172.21.15.129:0/2115522764 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4960039020 0x7f496003b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:59.065+0000 7f4979116700 1 --2- 172.21.15.129:0/2115522764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49741077a0 0x7f49740740e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:46:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:59.065+0000 7f4979116700 1 -- 172.21.15.129:0/2115522764 >> 172.21.15.129:0/2115522764 conn(0x7f49740fd2b0 msgr2=0x7f49741096b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:46:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:59.065+0000 7f4979116700 1 -- 172.21.15.129:0/2115522764 shutdown_connections 2024-03-11T20:46:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:46:59.065+0000 7f4979116700 1 -- 172.21.15.129:0/2115522764 wait complete. 2024-03-11T20:46:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:adjusted tunables profile to default 2024-03-11T20:46:59.665 INFO:tasks.cephadm:Adding mon.a on smithi129 2024-03-11T20:46:59.665 INFO:tasks.cephadm:Adding mon.c on smithi129 2024-03-11T20:46:59.665 INFO:tasks.cephadm:Adding mon.b on smithi157 2024-03-11T20:46:59.665 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi129:172.21.15.129=a;smithi129:[v2:172.21.15.129:3301,v1:172.21.15.129:6790]=c;smithi157:172.21.15.157=b' 2024-03-11T20:46:59.902 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:46:59.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:59 smithi129 ceph-mon[30184]: Updating smithi157:/etc/ceph/ceph.conf 2024-03-11T20:46:59.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:59 smithi129 ceph-mon[30184]: Updating smithi157:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:46:59.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:59 smithi129 ceph-mon[30184]: Updating smithi157:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:46:59.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:59 smithi129 ceph-mon[30184]: Updating smithi157:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:46:59.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:59 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:59.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:59 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:59.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:59 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:46:59.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2115522764' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T20:46:59.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:46:59 smithi129 ceph-mon[30184]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:47:00.927 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:00 smithi129 ceph-mon[30184]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:01.281 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.284+0000 7f6948f25700 1 -- 172.21.15.157:0/3167627875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6944107790 msgr2=0x7f6944107b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:01.281 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.284+0000 7f6948f25700 1 --2- 172.21.15.157:0/3167627875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6944107790 0x7f6944107b70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6934009910 tx=0x7f6934009c20 comp rx=0 tx=0).stop 2024-03-11T20:47:01.281 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.284+0000 7f6948f25700 1 -- 172.21.15.157:0/3167627875 shutdown_connections 2024-03-11T20:47:01.281 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.284+0000 7f6948f25700 1 --2- 172.21.15.157:0/3167627875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6944107790 0x7f6944107b70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:01.281 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.284+0000 7f6948f25700 1 -- 172.21.15.157:0/3167627875 >> 172.21.15.157:0/3167627875 conn(0x7f69440fd2c0 msgr2=0x7f69440ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:47:01.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.284+0000 7f6948f25700 1 -- 172.21.15.157:0/3167627875 shutdown_connections 2024-03-11T20:47:01.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.284+0000 7f6948f25700 1 -- 172.21.15.157:0/3167627875 wait complete. 2024-03-11T20:47:01.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.285+0000 7f6948f25700 1 Processor -- start 2024-03-11T20:47:01.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.285+0000 7f6948f25700 1 -- start start 2024-03-11T20:47:01.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.285+0000 7f6948f25700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6944107790 0x7f694419c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:01.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.285+0000 7f6948f25700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694410b250 con 0x7f6944107790 2024-03-11T20:47:01.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.286+0000 7f694259c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6944107790 0x7f694419c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:01.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.286+0000 7f694259c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6944107790 0x7f694419c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.157:42320/0 (socket says 172.21.15.157:42320) 2024-03-11T20:47:01.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.286+0000 7f694259c700 1 -- 172.21.15.157:0/3327980123 learned_addr learned my addr 172.21.15.157:0/3327980123 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:47:01.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.286+0000 7f694259c700 1 -- 172.21.15.157:0/3327980123 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694419cba0 con 0x7f6944107790 2024-03-11T20:47:01.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.286+0000 7f694259c700 1 --2- 172.21.15.157:0/3327980123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6944107790 0x7f694419c660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6934006010 tx=0x7f693400f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:47:01.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.286+0000 7f69337fe700 1 -- 172.21.15.157:0/3327980123 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6934027070 con 0x7f6944107790 2024-03-11T20:47:01.282 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.286+0000 7f6948f25700 1 -- 172.21.15.157:0/3327980123 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694419ce90 con 0x7f6944107790 2024-03-11T20:47:01.283 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.286+0000 7f69337fe700 1 -- 172.21.15.157:0/3327980123 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6934022470 con 0x7f6944107790 2024-03-11T20:47:01.283 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.286+0000 7f69337fe700 1 -- 172.21.15.157:0/3327980123 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6934017440 con 0x7f6944107790 2024-03-11T20:47:01.283 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.286+0000 7f6948f25700 1 -- 172.21.15.157:0/3327980123 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6944196900 con 0x7f6944107790 2024-03-11T20:47:01.283 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.287+0000 7f69337fe700 1 -- 172.21.15.157:0/3327980123 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f693402a070 con 0x7f6944107790 2024-03-11T20:47:01.284 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.287+0000 7f69337fe700 1 --2- 172.21.15.157:0/3327980123 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f692c038fc0 0x7f692c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:01.284 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.287+0000 7f6948f25700 1 -- 172.21.15.157:0/3327980123 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6944102ed0 con 0x7f6944107790 2024-03-11T20:47:01.284 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.287+0000 7f69337fe700 1 -- 172.21.15.157:0/3327980123 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f693401e030 con 0x7f6944107790 2024-03-11T20:47:01.284 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.288+0000 7f6941d9b700 1 --2- 172.21.15.157:0/3327980123 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f692c038fc0 0x7f692c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:01.284 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.288+0000 7f6941d9b700 1 --2- 172.21.15.157:0/3327980123 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f692c038fc0 0x7f692c03b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6938006fd0 tx=0x7f6938006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:47:01.288 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.293+0000 7f69337fe700 1 -- 172.21.15.157:0/3327980123 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f6934013070 con 0x7f6944107790 2024-03-11T20:47:01.497 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.502+0000 7f6948f25700 1 -- 172.21.15.157:0/3327980123 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi129:172.21.15.129=a;smithi129:[v2:172.21.15.129:3301,v1:172.21.15.129:6790]=c;smithi157:172.21.15.157=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f69440008d0 con 0x7f692c038fc0 2024-03-11T20:47:01.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.509+0000 7f69337fe700 1 -- 172.21.15.157:0/3327980123 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f69440008d0 con 0x7f692c038fc0 2024-03-11T20:47:01.504 INFO:teuthology.orchestra.run.smithi157.stdout:Scheduled mon update... 2024-03-11T20:47:01.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.511+0000 7f6948f25700 1 -- 172.21.15.157:0/3327980123 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f692c038fc0 msgr2=0x7f692c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:01.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.511+0000 7f6948f25700 1 --2- 172.21.15.157:0/3327980123 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f692c038fc0 0x7f692c03b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6938006fd0 tx=0x7f6938006e40 comp rx=0 tx=0).stop 2024-03-11T20:47:01.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.511+0000 7f6948f25700 1 -- 172.21.15.157:0/3327980123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6944107790 msgr2=0x7f694419c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:01.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.511+0000 7f6948f25700 1 --2- 172.21.15.157:0/3327980123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6944107790 0x7f694419c660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6934006010 tx=0x7f693400f740 comp rx=0 tx=0).stop 2024-03-11T20:47:01.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.511+0000 7f6948f25700 1 -- 172.21.15.157:0/3327980123 shutdown_connections 2024-03-11T20:47:01.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.511+0000 7f6948f25700 1 --2- 172.21.15.157:0/3327980123 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f692c038fc0 0x7f692c03b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:01.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.511+0000 7f6948f25700 1 --2- 172.21.15.157:0/3327980123 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6944107790 0x7f694419c660 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:01.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.511+0000 7f6948f25700 1 -- 172.21.15.157:0/3327980123 >> 172.21.15.157:0/3327980123 conn(0x7f69440fd2c0 msgr2=0x7f6944109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:47:01.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.511+0000 7f6948f25700 1 -- 172.21.15.157:0/3327980123 shutdown_connections 2024-03-11T20:47:01.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:01.511+0000 7f6948f25700 1 -- 172.21.15.157:0/3327980123 wait complete. 2024-03-11T20:47:02.115 DEBUG:teuthology.orchestra.run.smithi129:mon.c> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mon.c.service 2024-03-11T20:47:02.119 DEBUG:teuthology.orchestra.run.smithi157:mon.b> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mon.b.service 2024-03-11T20:47:02.121 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T20:47:02.121 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T20:47:02.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:02 smithi129 ceph-mon[30184]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi129:172.21.15.129=a;smithi129:[v2:172.21.15.129:3301,v1:172.21.15.129:6790]=c;smithi157:172.21.15.157=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:47:02.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:02 smithi129 ceph-mon[30184]: Saving service mon spec with placement smithi129:172.21.15.129=a;smithi129:[v2:172.21.15.129:3301,v1:172.21.15.129:6790]=c;smithi157:172.21.15.157=b;count:3 2024-03-11T20:47:02.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:02 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:02.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:02 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:02 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:02 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:02 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:02 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:47:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:02 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:02 smithi129 ceph-mon[30184]: Deploying daemon mon.b on smithi157 2024-03-11T20:47:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:02 smithi129 ceph-mon[30184]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:03.784 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:47:04.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:04 smithi129 ceph-mon[30184]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:06.365 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.189+0000 7f687e22e700 1 -- 172.21.15.157:0/1996471814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6878100090 msgr2=0x7f6878100470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:06.365 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.189+0000 7f687e22e700 1 --2- 172.21.15.157:0/1996471814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6878100090 0x7f6878100470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6868009910 tx=0x7f6868009c20 comp rx=0 tx=0).stop 2024-03-11T20:47:06.365 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.189+0000 7f687e22e700 1 -- 172.21.15.157:0/1996471814 shutdown_connections 2024-03-11T20:47:06.365 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.189+0000 7f687e22e700 1 --2- 172.21.15.157:0/1996471814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6878100090 0x7f6878100470 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:06.365 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.189+0000 7f687e22e700 1 -- 172.21.15.157:0/1996471814 >> 172.21.15.157:0/1996471814 conn(0x7f68780fb920 msgr2=0x7f68780fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:47:06.365 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.189+0000 7f687e22e700 1 -- 172.21.15.157:0/1996471814 shutdown_connections 2024-03-11T20:47:06.365 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.189+0000 7f687e22e700 1 -- 172.21.15.157:0/1996471814 wait complete. 2024-03-11T20:47:06.365 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.189+0000 7f687e22e700 1 Processor -- start 2024-03-11T20:47:06.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.190+0000 7f687e22e700 1 -- start start 2024-03-11T20:47:06.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.190+0000 7f687e22e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6878100090 0x7f687819c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:06.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.190+0000 7f687e22e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68781098d0 con 0x7f6878100090 2024-03-11T20:47:06.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.190+0000 7f68777fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6878100090 0x7f687819c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:06.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.190+0000 7f68777fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6878100090 0x7f687819c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.157:42334/0 (socket says 172.21.15.157:42334) 2024-03-11T20:47:06.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.190+0000 7f68777fe700 1 -- 172.21.15.157:0/1419677655 learned_addr learned my addr 172.21.15.157:0/1419677655 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:47:06.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.190+0000 7f68777fe700 1 -- 172.21.15.157:0/1419677655 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687819cb20 con 0x7f6878100090 2024-03-11T20:47:06.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.191+0000 7f68777fe700 1 --2- 172.21.15.157:0/1419677655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6878100090 0x7f687819c5e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6868006010 tx=0x7f686800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:47:06.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.191+0000 7f6874ff9700 1 -- 172.21.15.157:0/1419677655 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6868027070 con 0x7f6878100090 2024-03-11T20:47:06.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.191+0000 7f687e22e700 1 -- 172.21.15.157:0/1419677655 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f687819ce10 con 0x7f6878100090 2024-03-11T20:47:06.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.191+0000 7f6874ff9700 1 -- 172.21.15.157:0/1419677655 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6868022470 con 0x7f6878100090 2024-03-11T20:47:06.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.191+0000 7f687e22e700 1 -- 172.21.15.157:0/1419677655 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68781968d0 con 0x7f6878100090 2024-03-11T20:47:06.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.191+0000 7f6874ff9700 1 -- 172.21.15.157:0/1419677655 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6868017570 con 0x7f6878100090 2024-03-11T20:47:06.367 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.192+0000 7f6874ff9700 1 -- 172.21.15.157:0/1419677655 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f686802a070 con 0x7f6878100090 2024-03-11T20:47:06.367 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.192+0000 7f687e22e700 1 -- 172.21.15.157:0/1419677655 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6878101570 con 0x7f6878100090 2024-03-11T20:47:06.367 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.192+0000 7f6874ff9700 1 --2- 172.21.15.157:0/1419677655 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6860038fc0 0x7f686003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:06.367 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.192+0000 7f6874ff9700 1 -- 172.21.15.157:0/1419677655 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f686801e030 con 0x7f6878100090 2024-03-11T20:47:06.367 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.192+0000 7f6876ffd700 1 --2- 172.21.15.157:0/1419677655 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6860038fc0 0x7f686003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:06.367 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.193+0000 7f6876ffd700 1 --2- 172.21.15.157:0/1419677655 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6860038fc0 0x7f686003b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f686c006fd0 tx=0x7f686c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:47:06.367 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.198+0000 7f6874ff9700 1 -- 172.21.15.157:0/1419677655 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f6868013070 con 0x7f6878100090 2024-03-11T20:47:06.476 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.481+0000 7f687e22e700 1 -- 172.21.15.157:0/1419677655 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6878063410 con 0x7f6878100090 2024-03-11T20:47:06.476 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.481+0000 7f6874ff9700 1 -- 172.21.15.157:0/1419677655 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+749 (secure 0 0 0) 0x7f68680142b0 con 0x7f6878100090 2024-03-11T20:47:06.476 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:47:06.477 INFO:teuthology.orchestra.run.smithi157.stdout:{"epoch":1,"fsid":"27bbebfa-dfe8-11ee-95c8-87774f69a715","modified":"2024-03-11T20:45:31.660199Z","created":"2024-03-11T20:45:31.660199Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3300","nonce":0},{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T20:47:06.479 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.484+0000 7f687e22e700 1 -- 172.21.15.157:0/1419677655 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6860038fc0 msgr2=0x7f686003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:06.479 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.484+0000 7f687e22e700 1 --2- 172.21.15.157:0/1419677655 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6860038fc0 0x7f686003b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f686c006fd0 tx=0x7f686c006e40 comp rx=0 tx=0).stop 2024-03-11T20:47:06.479 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.484+0000 7f687e22e700 1 -- 172.21.15.157:0/1419677655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6878100090 msgr2=0x7f687819c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:06.479 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.484+0000 7f687e22e700 1 --2- 172.21.15.157:0/1419677655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6878100090 0x7f687819c5e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6868006010 tx=0x7f686800f740 comp rx=0 tx=0).stop 2024-03-11T20:47:06.479 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.484+0000 7f687e22e700 1 -- 172.21.15.157:0/1419677655 shutdown_connections 2024-03-11T20:47:06.479 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.484+0000 7f687e22e700 1 --2- 172.21.15.157:0/1419677655 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6860038fc0 0x7f686003b480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:06.479 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.484+0000 7f687e22e700 1 --2- 172.21.15.157:0/1419677655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6878100090 0x7f687819c5e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:06.479 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.484+0000 7f687e22e700 1 -- 172.21.15.157:0/1419677655 >> 172.21.15.157:0/1419677655 conn(0x7f68780fb920 msgr2=0x7f6878069400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:47:06.479 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.484+0000 7f687e22e700 1 -- 172.21.15.157:0/1419677655 shutdown_connections 2024-03-11T20:47:06.479 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:06.484+0000 7f687e22e700 1 -- 172.21.15.157:0/1419677655 wait complete. 2024-03-11T20:47:06.480 INFO:teuthology.orchestra.run.smithi157.stderr:dumped monmap epoch 1 2024-03-11T20:47:08.375 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T20:47:08.376 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T20:47:08.628 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:47:11.573 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.578+0000 7ff376159700 1 -- 172.21.15.157:0/3175783100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3701067e0 msgr2=0x7ff370106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:11.573 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.578+0000 7ff376159700 1 --2- 172.21.15.157:0/3175783100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3701067e0 0x7ff370106bc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff364005fa0 tx=0x7ff36400fe80 comp rx=0 tx=0).stop 2024-03-11T20:47:11.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.578+0000 7ff376159700 1 -- 172.21.15.157:0/3175783100 shutdown_connections 2024-03-11T20:47:11.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.578+0000 7ff376159700 1 --2- 172.21.15.157:0/3175783100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3701067e0 0x7ff370106bc0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:11.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.578+0000 7ff376159700 1 -- 172.21.15.157:0/3175783100 >> 172.21.15.157:0/3175783100 conn(0x7ff3700fc2f0 msgr2=0x7ff3700fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:47:11.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.578+0000 7ff376159700 1 -- 172.21.15.157:0/3175783100 shutdown_connections 2024-03-11T20:47:11.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.578+0000 7ff376159700 1 -- 172.21.15.157:0/3175783100 wait complete. 2024-03-11T20:47:11.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.579+0000 7ff376159700 1 Processor -- start 2024-03-11T20:47:11.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.579+0000 7ff376159700 1 -- start start 2024-03-11T20:47:11.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.579+0000 7ff376159700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3701067e0 0x7ff37019e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:11.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.579+0000 7ff376159700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff37019ee30 0x7ff370198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:11.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.579+0000 7ff376159700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37010a270 con 0x7ff37019ee30 2024-03-11T20:47:11.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.579+0000 7ff376159700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37010a0f0 con 0x7ff3701067e0 2024-03-11T20:47:11.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.579+0000 7ff36f7fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3701067e0 0x7ff37019e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:11.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.580+0000 7ff36f7fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3701067e0 0x7ff37019e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:42992/0 (socket says 172.21.15.157:42992) 2024-03-11T20:47:11.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.580+0000 7ff36f7fe700 1 -- 172.21.15.157:0/2203021410 learned_addr learned my addr 172.21.15.157:0/2203021410 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:47:11.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.580+0000 7ff36effd700 1 --2- 172.21.15.157:0/2203021410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff37019ee30 0x7ff370198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:11.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.580+0000 7ff36f7fe700 1 -- 172.21.15.157:0/2203021410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3701067e0 msgr2=0x7ff37019e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-11T20:47:11.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.580+0000 7ff36f7fe700 1 -- 172.21.15.157:0/2203021410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3701067e0 msgr2=0x7ff37019e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-11T20:47:11.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.580+0000 7ff36f7fe700 1 --2- 172.21.15.157:0/2203021410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3701067e0 0x7ff37019e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-11T20:47:11.577 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.580+0000 7ff36f7fe700 1 --2- 172.21.15.157:0/2203021410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3701067e0 0x7ff37019e8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T20:47:11.577 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.580+0000 7ff36effd700 1 -- 172.21.15.157:0/2203021410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3701067e0 msgr2=0x7ff37019e8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T20:47:11.577 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.580+0000 7ff36effd700 1 --2- 172.21.15.157:0/2203021410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3701067e0 0x7ff37019e8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:11.577 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.580+0000 7ff36effd700 1 -- 172.21.15.157:0/2203021410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37019f470 con 0x7ff37019ee30 2024-03-11T20:47:11.577 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.580+0000 7ff36effd700 1 --2- 172.21.15.157:0/2203021410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff37019ee30 0x7ff370198ab0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff36000b4d0 tx=0x7ff36000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:47:11.578 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.581+0000 7ff36cff9700 1 -- 172.21.15.157:0/2203021410 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7ff360011630 con 0x7ff37019ee30 2024-03-11T20:47:11.578 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.582+0000 7ff376159700 1 -- 172.21.15.157:0/2203021410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff370198ff0 con 0x7ff37019ee30 2024-03-11T20:47:11.578 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.582+0000 7ff36cff9700 1 -- 172.21.15.157:0/2203021410 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ff360011c50 con 0x7ff37019ee30 2024-03-11T20:47:11.579 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.582+0000 7ff36cff9700 1 -- 172.21.15.157:0/2203021410 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7ff360010c50 con 0x7ff37019ee30 2024-03-11T20:47:11.579 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.582+0000 7ff376159700 1 -- 172.21.15.157:0/2203021410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff370199580 con 0x7ff37019ee30 2024-03-11T20:47:11.579 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.582+0000 7ff36cff9700 1 -- 172.21.15.157:0/2203021410 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7ff360018020 con 0x7ff37019ee30 2024-03-11T20:47:11.579 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.583+0000 7ff36cff9700 1 --2- 172.21.15.157:0/2203021410 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff358038d80 0x7ff35803b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:11.579 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.583+0000 7ff36cff9700 1 -- 172.21.15.157:0/2203021410 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff36004c7f0 con 0x7ff37019ee30 2024-03-11T20:47:11.579 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.583+0000 7ff36f7fe700 1 --2- 172.21.15.157:0/2203021410 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff358038d80 0x7ff35803b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:11.579 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.583+0000 7ff376159700 1 -- 172.21.15.157:0/2203021410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff370101f20 con 0x7ff37019ee30 2024-03-11T20:47:11.579 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.583+0000 7ff36f7fe700 1 --2- 172.21.15.157:0/2203021410 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff358038d80 0x7ff35803b240 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff364005cd0 tx=0x7ff364005c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:47:11.583 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.588+0000 7ff36cff9700 1 -- 172.21.15.157:0/2203021410 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7ff36001f020 con 0x7ff37019ee30 2024-03-11T20:47:11.860 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.865+0000 7ff376159700 1 -- 172.21.15.157:0/2203021410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff370199fe0 con 0x7ff37019ee30 2024-03-11T20:47:11.860 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:11 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:11.861 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.865+0000 7ff36cff9700 1 -- 172.21.15.157:0/2203021410 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1019 (secure 0 0 0) 0x7ff360018570 con 0x7ff37019ee30 2024-03-11T20:47:11.861 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:47:11.861 INFO:teuthology.orchestra.run.smithi157.stdout:{"epoch":2,"fsid":"27bbebfa-dfe8-11ee-95c8-87774f69a715","modified":"2024-03-11T20:47:06.530715Z","created":"2024-03-11T20:45:31.660199Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3300","nonce":0},{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:3300","nonce":0},{"type":"v1","addr":"172.21.15.157:6789","nonce":0}]},"addr":"172.21.15.157:6789/0","public_addr":"172.21.15.157:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T20:47:11.863 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.868+0000 7ff376159700 1 -- 172.21.15.157:0/2203021410 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff358038d80 msgr2=0x7ff35803b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:11.863 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.868+0000 7ff376159700 1 --2- 172.21.15.157:0/2203021410 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff358038d80 0x7ff35803b240 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff364005cd0 tx=0x7ff364005c00 comp rx=0 tx=0).stop 2024-03-11T20:47:11.863 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.868+0000 7ff376159700 1 -- 172.21.15.157:0/2203021410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff37019ee30 msgr2=0x7ff370198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:11.863 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.868+0000 7ff376159700 1 --2- 172.21.15.157:0/2203021410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff37019ee30 0x7ff370198ab0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff36000b4d0 tx=0x7ff36000b890 comp rx=0 tx=0).stop 2024-03-11T20:47:11.863 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.868+0000 7ff376159700 1 -- 172.21.15.157:0/2203021410 shutdown_connections 2024-03-11T20:47:11.863 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.868+0000 7ff376159700 1 --2- 172.21.15.157:0/2203021410 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff358038d80 0x7ff35803b240 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:11.863 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.868+0000 7ff376159700 1 --2- 172.21.15.157:0/2203021410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3701067e0 0x7ff37019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:11.863 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.868+0000 7ff376159700 1 --2- 172.21.15.157:0/2203021410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff37019ee30 0x7ff370198ab0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:11.863 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.868+0000 7ff376159700 1 -- 172.21.15.157:0/2203021410 >> 172.21.15.157:0/2203021410 conn(0x7ff3700fc2f0 msgr2=0x7ff370108dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:47:11.863 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.868+0000 7ff376159700 1 -- 172.21.15.157:0/2203021410 shutdown_connections 2024-03-11T20:47:11.863 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:11.868+0000 7ff376159700 1 -- 172.21.15.157:0/2203021410 wait complete. 2024-03-11T20:47:11.864 INFO:teuthology.orchestra.run.smithi157.stderr:dumped monmap epoch 2 2024-03-11T20:47:11.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:47:11.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:11.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: mon.a calling monitor election 2024-03-11T20:47:11.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:11.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:11.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:11.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: mon.b calling monitor election 2024-03-11T20:47:11.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:11.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:11.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:11.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:11.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T20:47:11.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: monmap e2: 2 mons at {a=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],b=[v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:47:11.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: fsmap 2024-03-11T20:47:11.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:47:11.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: mgrmap e13: y(active, since 42s) 2024-03-11T20:47:11.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: overall HEALTH_OK 2024-03-11T20:47:11.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:11.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:11.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:11.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:47:11.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:12.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:12 smithi157 ceph-mon[33359]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:12.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:12 smithi157 ceph-mon[33359]: Deploying daemon mon.c on smithi129 2024-03-11T20:47:12.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.157:0/2203021410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T20:47:12.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:12 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:12.951 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:12 smithi129 ceph-mon[30184]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:12.951 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:12 smithi129 ceph-mon[30184]: Deploying daemon mon.c on smithi129 2024-03-11T20:47:12.952 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.157:0/2203021410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T20:47:12.952 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:12 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:13.459 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T20:47:13.460 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T20:47:13.705 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:47:14.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:14 smithi129 ceph-mon[30184]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:14.945 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:14 smithi157 ceph-mon[33359]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:15.196 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 podman[42028]: 2024-03-11 20:47:15.023907 +0000 UTC m=+0.121485592 container create 289a2a2bcc06b2926f01f6bca58af3fb23cb14d82e6bab49f32faeb2a0835d75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-11T20:47:15.271 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.273+0000 7fad2dc67700 1 -- 172.21.15.157:0/1816253674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad281067e0 msgr2=0x7fad28106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:15.271 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.273+0000 7fad2dc67700 1 --2- 172.21.15.157:0/1816253674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad281067e0 0x7fad28106bc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fad18009910 tx=0x7fad18009c20 comp rx=0 tx=0).stop 2024-03-11T20:47:15.271 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.273+0000 7fad2dc67700 1 -- 172.21.15.157:0/1816253674 shutdown_connections 2024-03-11T20:47:15.271 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.273+0000 7fad2dc67700 1 --2- 172.21.15.157:0/1816253674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad281067e0 0x7fad28106bc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:15.271 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.273+0000 7fad2dc67700 1 -- 172.21.15.157:0/1816253674 >> 172.21.15.157:0/1816253674 conn(0x7fad280fc2f0 msgr2=0x7fad280fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:47:15.271 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.274+0000 7fad2dc67700 1 -- 172.21.15.157:0/1816253674 shutdown_connections 2024-03-11T20:47:15.271 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.274+0000 7fad2dc67700 1 -- 172.21.15.157:0/1816253674 wait complete. 2024-03-11T20:47:15.271 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.274+0000 7fad2dc67700 1 Processor -- start 2024-03-11T20:47:15.271 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.274+0000 7fad2dc67700 1 -- start start 2024-03-11T20:47:15.271 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.274+0000 7fad2dc67700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad281067e0 0x7fad2819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:15.271 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.274+0000 7fad2dc67700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2819cd90 0x7fad28196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:15.271 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.274+0000 7fad2dc67700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2810a2a0 con 0x7fad281067e0 2024-03-11T20:47:15.272 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.274+0000 7fad2dc67700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2810a120 con 0x7fad2819cd90 2024-03-11T20:47:15.272 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad277fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad281067e0 0x7fad2819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:15.272 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad277fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad281067e0 0x7fad2819c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.157:59084/0 (socket says 172.21.15.157:59084) 2024-03-11T20:47:15.272 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad277fe700 1 -- 172.21.15.157:0/1022724030 learned_addr learned my addr 172.21.15.157:0/1022724030 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:47:15.272 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad1ffff700 1 --2- 172.21.15.157:0/1022724030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2819cd90 0x7fad28196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:15.272 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad1ffff700 1 -- 172.21.15.157:0/1022724030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2819cd90 msgr2=0x7fad28196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-11T20:47:15.272 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad1ffff700 1 -- 172.21.15.157:0/1022724030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2819cd90 msgr2=0x7fad28196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-11T20:47:15.272 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad1ffff700 1 --2- 172.21.15.157:0/1022724030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2819cd90 0x7fad28196a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-11T20:47:15.272 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad1ffff700 1 --2- 172.21.15.157:0/1022724030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2819cd90 0x7fad28196a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T20:47:15.272 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad277fe700 1 -- 172.21.15.157:0/1022724030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2819cd90 msgr2=0x7fad28196a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T20:47:15.272 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad277fe700 1 --2- 172.21.15.157:0/1022724030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2819cd90 0x7fad28196a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:15.272 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad277fe700 1 -- 172.21.15.157:0/1022724030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2819d3d0 con 0x7fad281067e0 2024-03-11T20:47:15.272 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad277fe700 1 --2- 172.21.15.157:0/1022724030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad281067e0 0x7fad2819c850 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fad18004f80 tx=0x7fad18005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:47:15.273 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad257fa700 1 -- 172.21.15.157:0/1022724030 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fad18005dc0 con 0x7fad281067e0 2024-03-11T20:47:15.273 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad257fa700 1 -- 172.21.15.157:0/1022724030 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fad1800bde0 con 0x7fad281067e0 2024-03-11T20:47:15.273 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad2dc67700 1 -- 172.21.15.157:0/1022724030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad28196fa0 con 0x7fad281067e0 2024-03-11T20:47:15.273 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad257fa700 1 -- 172.21.15.157:0/1022724030 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fad18022c60 con 0x7fad281067e0 2024-03-11T20:47:15.273 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.275+0000 7fad2dc67700 1 -- 172.21.15.157:0/1022724030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad28197530 con 0x7fad281067e0 2024-03-11T20:47:15.274 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.276+0000 7fad257fa700 1 -- 172.21.15.157:0/1022724030 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fad1802a070 con 0x7fad281067e0 2024-03-11T20:47:15.274 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.276+0000 7fad2dc67700 1 -- 172.21.15.157:0/1022724030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad28101f20 con 0x7fad281067e0 2024-03-11T20:47:15.274 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.276+0000 7fad257fa700 1 --2- 172.21.15.157:0/1022724030 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fad14038db0 0x7fad1403b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:15.274 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.276+0000 7fad257fa700 1 -- 172.21.15.157:0/1022724030 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fad1801e030 con 0x7fad281067e0 2024-03-11T20:47:15.274 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.276+0000 7fad1ffff700 1 --2- 172.21.15.157:0/1022724030 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fad14038db0 0x7fad1403b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:15.274 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.277+0000 7fad1ffff700 1 --2- 172.21.15.157:0/1022724030 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fad14038db0 0x7fad1403b270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fad10009850 tx=0x7fad10008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:47:15.277 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.282+0000 7fad257fa700 1 -- 172.21.15.157:0/1022724030 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fad18013070 con 0x7fad281067e0 2024-03-11T20:47:15.422 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.426+0000 7fad257fa700 1 -- 172.21.15.157:0/1022724030 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1802b730 con 0x7fad281067e0 2024-03-11T20:47:15.547 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:15.552+0000 7fad2dc67700 1 -- 172.21.15.157:0/1022724030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad28197840 con 0x7fad281067e0 2024-03-11T20:47:15.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 podman[42028]: 2024-03-11 20:47:15.360809959 +0000 UTC m=+0.458388550 container init 289a2a2bcc06b2926f01f6bca58af3fb23cb14d82e6bab49f32faeb2a0835d75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-11T20:47:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 podman[42028]: 2024-03-11 20:47:15.367182178 +0000 UTC m=+0.464760769 container start 289a2a2bcc06b2926f01f6bca58af3fb23cb14d82e6bab49f32faeb2a0835d75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-11T20:47:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T20:47:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 7 2024-03-11T20:47:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: pidfile_write: ignore empty --pid-file 2024-03-11T20:47:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: load: jerasure load: lrc 2024-03-11T20:47:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: RocksDB version: 7.9.2 2024-03-11T20:47:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Git sha 0 2024-03-11T20:47:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-11T20:47:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: DB SUMMARY 2024-03-11T20:47:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: DB Session ID: FN6I0ZQZFTDHZAVBHJ29 2024-03-11T20:47:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: CURRENT file: CURRENT 2024-03-11T20:47:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T20:47:15.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-11T20:47:15.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-11T20:47:15.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-11T20:47:15.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.error_if_exists: 0 2024-03-11T20:47:15.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.create_if_missing: 0 2024-03-11T20:47:15.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.paranoid_checks: 1 2024-03-11T20:47:15.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T20:47:15.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T20:47:15.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T20:47:15.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.env: 0x562ab5841b20 2024-03-11T20:47:15.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T20:47:15.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.info_log: 0x562ab6dcb8e0 2024-03-11T20:47:15.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T20:47:15.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.statistics: (nil) 2024-03-11T20:47:15.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.use_fsync: 0 2024-03-11T20:47:15.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_log_file_size: 0 2024-03-11T20:47:15.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T20:47:15.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T20:47:15.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T20:47:15.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T20:47:15.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.allow_fallocate: 1 2024-03-11T20:47:15.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T20:47:15.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T20:47:15.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.use_direct_reads: 0 2024-03-11T20:47:15.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T20:47:15.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T20:47:15.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.db_log_dir: 2024-03-11T20:47:15.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.wal_dir: 2024-03-11T20:47:15.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T20:47:15.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T20:47:15.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T20:47:15.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T20:47:15.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T20:47:15.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T20:47:15.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T20:47:15.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T20:47:15.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.write_buffer_manager: 0x562ab6d9f400 2024-03-11T20:47:15.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T20:47:15.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T20:47:15.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T20:47:15.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T20:47:15.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T20:47:15.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T20:47:15.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T20:47:15.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T20:47:15.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.unordered_write: 0 2024-03-11T20:47:15.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T20:47:15.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T20:47:15.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T20:47:15.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T20:47:15.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.row_cache: None 2024-03-11T20:47:15.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.wal_filter: None 2024-03-11T20:47:15.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T20:47:15.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T20:47:15.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.two_write_queues: 0 2024-03-11T20:47:15.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T20:47:15.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.wal_compression: 0 2024-03-11T20:47:15.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.atomic_flush: 0 2024-03-11T20:47:15.735 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T20:47:15.735 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T20:47:15.735 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T20:47:15.735 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.log_readahead_size: 0 2024-03-11T20:47:15.735 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T20:47:15.735 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T20:47:15.735 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T20:47:15.735 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T20:47:15.735 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T20:47:15.735 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T20:47:15.735 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T20:47:15.735 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_background_jobs: 2 2024-03-11T20:47:15.735 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_background_compactions: -1 2024-03-11T20:47:15.736 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_subcompactions: 1 2024-03-11T20:47:15.736 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T20:47:15.736 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T20:47:15.736 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T20:47:15.736 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T20:47:15.736 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T20:47:15.736 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T20:47:15.736 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T20:47:15.736 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T20:47:15.736 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_open_files: -1 2024-03-11T20:47:15.736 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T20:47:15.736 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T20:47:15.736 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T20:47:15.736 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T20:47:15.737 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_background_flushes: -1 2024-03-11T20:47:15.737 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Compression algorithms supported: 2024-03-11T20:47:15.737 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T20:47:15.737 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: kZSTD supported: 0 2024-03-11T20:47:15.737 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: kXpressCompression supported: 0 2024-03-11T20:47:15.737 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T20:47:15.737 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: kZlibCompression supported: 1 2024-03-11T20:47:15.737 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: kSnappyCompression supported: 1 2024-03-11T20:47:15.737 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: kLZ4Compression supported: 1 2024-03-11T20:47:15.737 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: kBZip2Compression supported: 0 2024-03-11T20:47:15.737 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T20:47:15.737 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T20:47:15.737 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-11T20:47:15.737 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T20:47:15.737 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T20:47:15.738 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.merge_operator: 2024-03-11T20:47:15.738 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compaction_filter: None 2024-03-11T20:47:15.738 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T20:47:15.738 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T20:47:15.738 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T20:47:15.738 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T20:47:15.738 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562ab6dcb9e0) 2024-03-11T20:47:15.738 INFO:journalctl@ceph.mon.c.smithi129.stdout: cache_index_and_filter_blocks: 1 2024-03-11T20:47:15.738 INFO:journalctl@ceph.mon.c.smithi129.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T20:47:15.738 INFO:journalctl@ceph.mon.c.smithi129.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T20:47:15.738 INFO:journalctl@ceph.mon.c.smithi129.stdout: pin_top_level_index_and_filter: 1 2024-03-11T20:47:15.738 INFO:journalctl@ceph.mon.c.smithi129.stdout: index_type: 0 2024-03-11T20:47:15.738 INFO:journalctl@ceph.mon.c.smithi129.stdout: data_block_index_type: 0 2024-03-11T20:47:15.738 INFO:journalctl@ceph.mon.c.smithi129.stdout: index_shortening: 1 2024-03-11T20:47:15.739 INFO:journalctl@ceph.mon.c.smithi129.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T20:47:15.739 INFO:journalctl@ceph.mon.c.smithi129.stdout: checksum: 4 2024-03-11T20:47:15.739 INFO:journalctl@ceph.mon.c.smithi129.stdout: no_block_cache: 0 2024-03-11T20:47:15.739 INFO:journalctl@ceph.mon.c.smithi129.stdout: block_cache: 0x562ab6dd1090 2024-03-11T20:47:15.739 INFO:journalctl@ceph.mon.c.smithi129.stdout: block_cache_name: BinnedLRUCache 2024-03-11T20:47:15.739 INFO:journalctl@ceph.mon.c.smithi129.stdout: block_cache_options: 2024-03-11T20:47:15.739 INFO:journalctl@ceph.mon.c.smithi129.stdout: capacity : 536870912 2024-03-11T20:47:15.739 INFO:journalctl@ceph.mon.c.smithi129.stdout: num_shard_bits : 4 2024-03-11T20:47:15.739 INFO:journalctl@ceph.mon.c.smithi129.stdout: strict_capacity_limit : 0 2024-03-11T20:47:15.739 INFO:journalctl@ceph.mon.c.smithi129.stdout: high_pri_pool_ratio: 0.000 2024-03-11T20:47:15.739 INFO:journalctl@ceph.mon.c.smithi129.stdout: block_cache_compressed: (nil) 2024-03-11T20:47:15.739 INFO:journalctl@ceph.mon.c.smithi129.stdout: persistent_cache: (nil) 2024-03-11T20:47:15.739 INFO:journalctl@ceph.mon.c.smithi129.stdout: block_size: 4096 2024-03-11T20:47:15.740 INFO:journalctl@ceph.mon.c.smithi129.stdout: block_size_deviation: 10 2024-03-11T20:47:15.740 INFO:journalctl@ceph.mon.c.smithi129.stdout: block_restart_interval: 16 2024-03-11T20:47:15.740 INFO:journalctl@ceph.mon.c.smithi129.stdout: index_block_restart_interval: 1 2024-03-11T20:47:15.740 INFO:journalctl@ceph.mon.c.smithi129.stdout: metadata_block_size: 4096 2024-03-11T20:47:15.740 INFO:journalctl@ceph.mon.c.smithi129.stdout: partition_filters: 0 2024-03-11T20:47:15.740 INFO:journalctl@ceph.mon.c.smithi129.stdout: use_delta_encoding: 1 2024-03-11T20:47:15.740 INFO:journalctl@ceph.mon.c.smithi129.stdout: filter_policy: bloomfilter 2024-03-11T20:47:15.740 INFO:journalctl@ceph.mon.c.smithi129.stdout: whole_key_filtering: 1 2024-03-11T20:47:15.740 INFO:journalctl@ceph.mon.c.smithi129.stdout: verify_compression: 0 2024-03-11T20:47:15.740 INFO:journalctl@ceph.mon.c.smithi129.stdout: read_amp_bytes_per_bit: 0 2024-03-11T20:47:15.740 INFO:journalctl@ceph.mon.c.smithi129.stdout: format_version: 5 2024-03-11T20:47:15.740 INFO:journalctl@ceph.mon.c.smithi129.stdout: enable_index_compression: 1 2024-03-11T20:47:15.740 INFO:journalctl@ceph.mon.c.smithi129.stdout: block_align: 0 2024-03-11T20:47:15.740 INFO:journalctl@ceph.mon.c.smithi129.stdout: max_auto_readahead_size: 262144 2024-03-11T20:47:15.741 INFO:journalctl@ceph.mon.c.smithi129.stdout: prepopulate_block_cache: 0 2024-03-11T20:47:15.741 INFO:journalctl@ceph.mon.c.smithi129.stdout: initial_auto_readahead_size: 8192 2024-03-11T20:47:15.741 INFO:journalctl@ceph.mon.c.smithi129.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T20:47:15.741 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T20:47:15.741 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T20:47:15.741 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compression: NoCompression 2024-03-11T20:47:15.741 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T20:47:15.741 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T20:47:15.741 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T20:47:15.741 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.num_levels: 7 2024-03-11T20:47:15.741 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T20:47:15.741 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T20:47:15.741 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T20:47:15.742 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T20:47:15.742 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T20:47:15.742 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T20:47:15.742 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T20:47:15.742 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T20:47:15.742 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T20:47:15.742 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T20:47:15.742 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T20:47:15.742 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T20:47:15.742 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T20:47:15.742 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T20:47:15.742 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T20:47:15.742 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T20:47:15.743 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T20:47:15.743 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T20:47:15.743 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T20:47:15.743 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T20:47:15.743 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T20:47:15.743 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T20:47:15.743 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T20:47:15.743 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T20:47:15.743 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T20:47:15.743 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T20:47:15.743 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T20:47:15.743 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T20:47:15.743 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T20:47:15.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T20:47:15.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T20:47:15.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T20:47:15.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T20:47:15.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T20:47:15.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T20:47:15.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T20:47:15.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T20:47:15.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T20:47:15.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T20:47:15.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T20:47:15.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T20:47:15.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T20:47:15.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T20:47:15.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T20:47:15.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T20:47:15.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T20:47:15.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T20:47:15.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T20:47:15.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T20:47:15.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T20:47:15.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T20:47:15.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T20:47:15.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T20:47:15.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.table_properties_collectors: 2024-03-11T20:47:15.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.inplace_update_support: 0 2024-03-11T20:47:15.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T20:47:15.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T20:47:15.746 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T20:47:15.746 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T20:47:15.746 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.bloom_locality: 0 2024-03-11T20:47:15.746 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.max_successive_merges: 0 2024-03-11T20:47:15.746 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T20:47:15.746 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T20:47:15.746 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T20:47:15.746 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T20:47:15.746 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.ttl: 2592000 2024-03-11T20:47:15.746 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T20:47:15.746 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T20:47:15.746 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T20:47:15.746 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.enable_blob_files: false 2024-03-11T20:47:15.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.min_blob_size: 0 2024-03-11T20:47:15.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T20:47:15.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T20:47:15.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T20:47:15.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T20:47:15.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T20:47:15.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T20:47:15.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T20:47:15.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T20:47:15.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T20:47:15.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T20:47:15.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5a970057-3a38-4828-a819-a6ddd875cd82 2024-03-11T20:47:15.747 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710190035396897, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T20:47:15.749 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T20:47:15.749 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710190035397300, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710190035, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5a970057-3a38-4828-a819-a6ddd875cd82", "db_session_id": "FN6I0ZQZFTDHZAVBHJ29", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T20:47:15.749 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710190035397382, "job": 1, "event": "recovery_finished"} 2024-03-11T20:47:15.749 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T20:47:15.749 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T20:47:15.749 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562ab6ec4000 2024-03-11T20:47:15.749 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: DB pointer 0x562ab6eae000 2024-03-11T20:47:15.750 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T20:47:15.750 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T20:47:15.750 INFO:journalctl@ceph.mon.c.smithi129.stdout: ** DB Stats ** 2024-03-11T20:47:15.750 INFO:journalctl@ceph.mon.c.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T20:47:15.750 INFO:journalctl@ceph.mon.c.smithi129.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T20:47:15.750 INFO:journalctl@ceph.mon.c.smithi129.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T20:47:15.750 INFO:journalctl@ceph.mon.c.smithi129.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T20:47:15.750 INFO:journalctl@ceph.mon.c.smithi129.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T20:47:15.750 INFO:journalctl@ceph.mon.c.smithi129.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T20:47:15.750 INFO:journalctl@ceph.mon.c.smithi129.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T20:47:15.750 INFO:journalctl@ceph.mon.c.smithi129.stdout: 2024-03-11T20:47:15.750 INFO:journalctl@ceph.mon.c.smithi129.stdout: ** Compaction Stats [default] ** 2024-03-11T20:47:15.751 INFO:journalctl@ceph.mon.c.smithi129.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T20:47:15.751 INFO:journalctl@ceph.mon.c.smithi129.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T20:47:15.751 INFO:journalctl@ceph.mon.c.smithi129.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T20:47:15.751 INFO:journalctl@ceph.mon.c.smithi129.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T20:47:15.751 INFO:journalctl@ceph.mon.c.smithi129.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T20:47:15.751 INFO:journalctl@ceph.mon.c.smithi129.stdout: 2024-03-11T20:47:15.751 INFO:journalctl@ceph.mon.c.smithi129.stdout: ** Compaction Stats [default] ** 2024-03-11T20:47:15.751 INFO:journalctl@ceph.mon.c.smithi129.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T20:47:15.751 INFO:journalctl@ceph.mon.c.smithi129.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T20:47:15.751 INFO:journalctl@ceph.mon.c.smithi129.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T20:47:15.751 INFO:journalctl@ceph.mon.c.smithi129.stdout: 2024-03-11T20:47:15.751 INFO:journalctl@ceph.mon.c.smithi129.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T20:47:15.751 INFO:journalctl@ceph.mon.c.smithi129.stdout: 2024-03-11T20:47:15.751 INFO:journalctl@ceph.mon.c.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T20:47:15.752 INFO:journalctl@ceph.mon.c.smithi129.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T20:47:15.752 INFO:journalctl@ceph.mon.c.smithi129.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T20:47:15.752 INFO:journalctl@ceph.mon.c.smithi129.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T20:47:15.752 INFO:journalctl@ceph.mon.c.smithi129.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T20:47:15.752 INFO:journalctl@ceph.mon.c.smithi129.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T20:47:15.752 INFO:journalctl@ceph.mon.c.smithi129.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T20:47:15.752 INFO:journalctl@ceph.mon.c.smithi129.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T20:47:15.752 INFO:journalctl@ceph.mon.c.smithi129.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T20:47:15.752 INFO:journalctl@ceph.mon.c.smithi129.stdout: Block cache BinnedLRUCache@0x562ab6dd1090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-11T20:47:15.752 INFO:journalctl@ceph.mon.c.smithi129.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T20:47:15.752 INFO:journalctl@ceph.mon.c.smithi129.stdout: 2024-03-11T20:47:15.752 INFO:journalctl@ceph.mon.c.smithi129.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T20:47:15.752 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-11T20:47:15.752 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: using public_addrv [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2024-03-11T20:47:15.752 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: starting mon.c rank -1 at public addrs [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] at bind addrs [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:47:15.753 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.c@-1(???) e0 preinit fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:47:15.753 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.c@-1(synchronizing).mds e1 new map 2024-03-11T20:47:15.753 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-11T20:47:15.753 INFO:journalctl@ceph.mon.c.smithi129.stdout: e1 2024-03-11T20:47:15.753 INFO:journalctl@ceph.mon.c.smithi129.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T20:47:15.753 INFO:journalctl@ceph.mon.c.smithi129.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T20:47:15.753 INFO:journalctl@ceph.mon.c.smithi129.stdout: legacy client fscid: -1 2024-03-11T20:47:15.753 INFO:journalctl@ceph.mon.c.smithi129.stdout: 2024-03-11T20:47:15.753 INFO:journalctl@ceph.mon.c.smithi129.stdout: No filesystems configured 2024-03-11T20:47:15.753 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T20:47:15.753 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T20:47:15.753 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T20:47:15.753 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T20:47:15.753 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T20:47:15.753 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T20:47:15.754 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T20:47:15.754 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T20:47:15.754 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T20:47:15.754 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T20:47:15.754 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:15.754 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:47:15.754 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:15.754 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:15.754 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:15.754 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:15.754 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:15.754 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:15.754 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi157", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:47:15.755 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:15.755 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:15.755 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2115522764' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T20:47:15.755 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: Updating smithi157:/etc/ceph/ceph.conf 2024-03-11T20:47:15.755 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: Updating smithi157:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:47:15.755 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: Updating smithi157:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:47:15.755 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: Updating smithi157:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:47:15.755 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:15.755 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:15.755 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:15.755 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2115522764' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T20:47:15.755 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:47:15.755 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:15.755 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi129:172.21.15.129=a;smithi129:[v2:172.21.15.129:3301,v1:172.21.15.129:6790]=c;smithi157:172.21.15.157=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:47:15.755 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: Saving service mon spec with placement smithi129:172.21.15.129=a;smithi129:[v2:172.21.15.129:3301,v1:172.21.15.129:6790]=c;smithi157:172.21.15.157=b;count:3 2024-03-11T20:47:15.756 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:15.756 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:15.756 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:15.756 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:15.756 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:15.756 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:47:15.756 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:15.756 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: Deploying daemon mon.b on smithi157 2024-03-11T20:47:15.756 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:15.756 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:15.756 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:47:15.756 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:15.756 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.a calling monitor election 2024-03-11T20:47:15.756 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:15.757 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:15.757 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:15.757 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.b calling monitor election 2024-03-11T20:47:15.757 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:15.757 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:15.757 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:15.757 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:15.757 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T20:47:15.757 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: monmap e2: 2 mons at {a=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],b=[v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:47:15.757 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: fsmap 2024-03-11T20:47:15.757 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:47:15.757 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mgrmap e13: y(active, since 42s) 2024-03-11T20:47:15.757 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: overall HEALTH_OK 2024-03-11T20:47:15.757 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:15.758 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:15.758 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:15.758 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:47:15.758 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:15.758 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:15.758 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: Deploying daemon mon.c on smithi129 2024-03-11T20:47:15.758 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.157:0/2203021410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T20:47:15.758 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:15.758 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:15.758 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 ceph-mon[42066]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-11T20:47:15.758 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 bash[42028]: 289a2a2bcc06b2926f01f6bca58af3fb23cb14d82e6bab49f32faeb2a0835d75 2024-03-11T20:47:15.758 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:15 smithi129 systemd[1]: Started Ceph mon.c for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T20:47:20.454 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:20.458+0000 7fad257fa700 1 -- 172.21.15.157:0/1022724030 <== mon.0 v2:172.21.15.129:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1287 (secure 0 0 0) 0x7fad18031020 con 0x7fad281067e0 2024-03-11T20:47:20.454 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-11T20:47:20.455 INFO:teuthology.orchestra.run.smithi157.stdout:{"epoch":3,"fsid":"27bbebfa-dfe8-11ee-95c8-87774f69a715","modified":"2024-03-11T20:47:15.426901Z","created":"2024-03-11T20:45:31.660199Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3300","nonce":0},{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:3300","nonce":0},{"type":"v1","addr":"172.21.15.157:6789","nonce":0}]},"addr":"172.21.15.157:6789/0","public_addr":"172.21.15.157:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3301","nonce":0},{"type":"v1","addr":"172.21.15.129:6790","nonce":0}]},"addr":"172.21.15.129:6790/0","public_addr":"172.21.15.129:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T20:47:20.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:20.461+0000 7fad2dc67700 1 -- 172.21.15.157:0/1022724030 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fad14038db0 msgr2=0x7fad1403b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:20.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:20.461+0000 7fad2dc67700 1 --2- 172.21.15.157:0/1022724030 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fad14038db0 0x7fad1403b270 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fad10009850 tx=0x7fad10008040 comp rx=0 tx=0).stop 2024-03-11T20:47:20.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:20.461+0000 7fad2dc67700 1 -- 172.21.15.157:0/1022724030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad281067e0 msgr2=0x7fad2819c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:20.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:20.461+0000 7fad2dc67700 1 --2- 172.21.15.157:0/1022724030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad281067e0 0x7fad2819c850 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fad18004f80 tx=0x7fad18005060 comp rx=0 tx=0).stop 2024-03-11T20:47:20.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:20.461+0000 7fad2dc67700 1 -- 172.21.15.157:0/1022724030 shutdown_connections 2024-03-11T20:47:20.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:20.461+0000 7fad2dc67700 1 --2- 172.21.15.157:0/1022724030 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fad14038db0 0x7fad1403b270 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:20.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:20.461+0000 7fad2dc67700 1 --2- 172.21.15.157:0/1022724030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad281067e0 0x7fad2819c850 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:20.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:20.461+0000 7fad2dc67700 1 --2- 172.21.15.157:0/1022724030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2819cd90 0x7fad28196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:20.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:20.461+0000 7fad2dc67700 1 -- 172.21.15.157:0/1022724030 >> 172.21.15.157:0/1022724030 conn(0x7fad280fc2f0 msgr2=0x7fad28069fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:47:20.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:20.462+0000 7fad2dc67700 1 -- 172.21.15.157:0/1022724030 shutdown_connections 2024-03-11T20:47:20.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:20.462+0000 7fad2dc67700 1 -- 172.21.15.157:0/1022724030 wait complete. 2024-03-11T20:47:20.458 INFO:teuthology.orchestra.run.smithi157.stderr:dumped monmap epoch 3 2024-03-11T20:47:20.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:47:20.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: mon.a calling monitor election 2024-03-11T20:47:20.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: mon.b calling monitor election 2024-03-11T20:47:20.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:20.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:20.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.157:0/1022724030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T20:47:20.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:20.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:20.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:20.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:20.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:20.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:20.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:20.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:20.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T20:47:20.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: monmap e3: 3 mons at {a=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],b=[v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0],c=[v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:47:20.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: fsmap 2024-03-11T20:47:20.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:47:20.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: mgrmap e13: y(active, since 50s) 2024-03-11T20:47:20.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-03-11T20:47:20.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-03-11T20:47:20.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-03-11T20:47:20.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: mon.c (rank 2) addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] is down (out of quorum) 2024-03-11T20:47:20.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:20.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:20.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:20.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:20.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:47:20.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: mon.a calling monitor election 2024-03-11T20:47:20.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: mon.b calling monitor election 2024-03-11T20:47:20.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:20.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:20.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.157:0/1022724030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T20:47:20.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:20.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:20.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:20.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T20:47:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: monmap e3: 3 mons at {a=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],b=[v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0],c=[v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:47:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: fsmap 2024-03-11T20:47:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:47:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: mgrmap e13: y(active, since 50s) 2024-03-11T20:47:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-03-11T20:47:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-03-11T20:47:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-03-11T20:47:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: mon.c (rank 2) addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] is down (out of quorum) 2024-03-11T20:47:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:20.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:20.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:20.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:21.037 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T20:47:21.037 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T20:47:21.550 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:21 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:21.550 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:21 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:21.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:21 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:21.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:21 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:21.893 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:47:22.537 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:47:22.537 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: mon.a calling monitor election 2024-03-11T20:47:22.537 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: mon.b calling monitor election 2024-03-11T20:47:22.537 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:47:22.537 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:22.537 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.157:0/1022724030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T20:47:22.538 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:22.538 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:22.538 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:22.538 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:22.538 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:22.538 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:22.538 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:22.539 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:22.539 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T20:47:22.539 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: monmap e3: 3 mons at {a=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],b=[v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0],c=[v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:47:22.539 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: fsmap 2024-03-11T20:47:22.539 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:47:22.539 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: mgrmap e13: y(active, since 50s) 2024-03-11T20:47:22.539 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-03-11T20:47:22.540 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-03-11T20:47:22.540 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-03-11T20:47:22.540 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: mon.c (rank 2) addr [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] is down (out of quorum) 2024-03-11T20:47:22.540 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:22.540 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:22.540 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:22.540 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:22.540 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:22.541 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:23.223 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: mon.c calling monitor election 2024-03-11T20:47:23.224 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-11T20:47:23.224 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: Updating smithi157:/etc/ceph/ceph.conf 2024-03-11T20:47:23.224 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: Updating smithi129:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:47:23.224 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: Updating smithi157:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:47:23.224 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: mon.c calling monitor election 2024-03-11T20:47:23.224 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: mon.b calling monitor election 2024-03-11T20:47:23.224 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: mon.a calling monitor election 2024-03-11T20:47:23.224 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T20:47:23.224 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: monmap e3: 3 mons at {a=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],b=[v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0],c=[v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:47:23.224 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: fsmap 2024-03-11T20:47:23.224 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:47:23.224 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: mgrmap e13: y(active, since 52s) 2024-03-11T20:47:23.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-03-11T20:47:23.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: Cluster is now healthy 2024-03-11T20:47:23.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: overall HEALTH_OK 2024-03-11T20:47:23.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T20:47:23.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:47:23.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:47:23.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:23.226 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[30184]: Reconfiguring daemon mon.a on smithi129 2024-03-11T20:47:23.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: mon.c calling monitor election 2024-03-11T20:47:23.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-11T20:47:23.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: Updating smithi157:/etc/ceph/ceph.conf 2024-03-11T20:47:23.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: Updating smithi129:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:47:23.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: Updating smithi157:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:47:23.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: mon.c calling monitor election 2024-03-11T20:47:23.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: mon.b calling monitor election 2024-03-11T20:47:23.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: mon.a calling monitor election 2024-03-11T20:47:23.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T20:47:23.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: monmap e3: 3 mons at {a=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],b=[v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0],c=[v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:47:23.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: fsmap 2024-03-11T20:47:23.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:47:23.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: mgrmap e13: y(active, since 52s) 2024-03-11T20:47:23.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-03-11T20:47:23.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: Cluster is now healthy 2024-03-11T20:47:23.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: overall HEALTH_OK 2024-03-11T20:47:23.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T20:47:23.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:47:23.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:47:23.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:23.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:23 smithi129 ceph-mon[42066]: Reconfiguring daemon mon.a on smithi129 2024-03-11T20:47:23.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: mon.c calling monitor election 2024-03-11T20:47:23.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-11T20:47:23.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: Updating smithi157:/etc/ceph/ceph.conf 2024-03-11T20:47:23.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: Updating smithi129:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:47:23.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: Updating smithi157:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:47:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: mon.c calling monitor election 2024-03-11T20:47:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: mon.b calling monitor election 2024-03-11T20:47:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: mon.a calling monitor election 2024-03-11T20:47:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T20:47:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: monmap e3: 3 mons at {a=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],b=[v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0],c=[v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T20:47:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: fsmap 2024-03-11T20:47:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T20:47:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: mgrmap e13: y(active, since 52s) 2024-03-11T20:47:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-03-11T20:47:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: Cluster is now healthy 2024-03-11T20:47:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: overall HEALTH_OK 2024-03-11T20:47:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:23.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T20:47:23.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:47:23.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:47:23.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:23.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:23 smithi157 ceph-mon[33359]: Reconfiguring daemon mon.a on smithi129 2024-03-11T20:47:24.227 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:24 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:24.227 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:24 smithi129 ceph-mon[30184]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:24.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:24 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:24.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:24 smithi129 ceph-mon[42066]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:24.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.362+0000 7f24b5546700 1 -- 172.21.15.129:0/1532455672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24b01066d0 msgr2=0x7f24b0106ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:24.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.362+0000 7f24b5546700 1 --2- 172.21.15.129:0/1532455672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24b01066d0 0x7f24b0106ab0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f24a4009910 tx=0x7f24a4009c20 comp rx=0 tx=0).stop 2024-03-11T20:47:24.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.362+0000 7f24b5546700 1 -- 172.21.15.129:0/1532455672 shutdown_connections 2024-03-11T20:47:24.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.362+0000 7f24b5546700 1 --2- 172.21.15.129:0/1532455672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24b01066d0 0x7f24b0106ab0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:24.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.362+0000 7f24b5546700 1 -- 172.21.15.129:0/1532455672 >> 172.21.15.129:0/1532455672 conn(0x7f24b00fc1e0 msgr2=0x7f24b00fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:47:24.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.362+0000 7f24b5546700 1 -- 172.21.15.129:0/1532455672 shutdown_connections 2024-03-11T20:47:24.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.362+0000 7f24b5546700 1 -- 172.21.15.129:0/1532455672 wait complete. 2024-03-11T20:47:24.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.363+0000 7f24b5546700 1 Processor -- start 2024-03-11T20:47:24.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.363+0000 7f24b5546700 1 -- start start 2024-03-11T20:47:24.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.363+0000 7f24b5546700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24b01066d0 0x7f24b019b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:24.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.363+0000 7f24b5546700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24b019bc00 0x7f24b0195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:24.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.363+0000 7f24b5546700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24b0195d80 0x7f24b0196230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:24.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.363+0000 7f24b5546700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b010a130 con 0x7f24b019bc00 2024-03-11T20:47:24.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.363+0000 7f24b5546700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b0109fb0 con 0x7f24b0195d80 2024-03-11T20:47:24.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.363+0000 7f24b5546700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b010a2b0 con 0x7f24b01066d0 2024-03-11T20:47:24.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.363+0000 7f24aeffd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24b01066d0 0x7f24b019b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:24.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.363+0000 7f24aeffd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24b01066d0 0x7f24b019b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:41422/0 (socket says 172.21.15.129:41422) 2024-03-11T20:47:24.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.363+0000 7f24aeffd700 1 -- 172.21.15.129:0/601199900 learned_addr learned my addr 172.21.15.129:0/601199900 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:47:24.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.363+0000 7f24ae7fc700 1 --2- 172.21.15.129:0/601199900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24b019bc00 0x7f24b0195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:24.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.364+0000 7f24af7fe700 1 --2- 172.21.15.129:0/601199900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24b0195d80 0x7f24b0196230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:24.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.364+0000 7f24aeffd700 1 -- 172.21.15.129:0/601199900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24b0195d80 msgr2=0x7f24b0196230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:24.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.364+0000 7f24aeffd700 1 --2- 172.21.15.129:0/601199900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24b0195d80 0x7f24b0196230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:24.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.364+0000 7f24aeffd700 1 -- 172.21.15.129:0/601199900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24b019bc00 msgr2=0x7f24b0195840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:24.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.364+0000 7f24aeffd700 1 --2- 172.21.15.129:0/601199900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24b019bc00 0x7f24b0195840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:24.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.364+0000 7f24aeffd700 1 -- 172.21.15.129:0/601199900 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b0069bb0 con 0x7f24b01066d0 2024-03-11T20:47:24.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.364+0000 7f24af7fe700 1 --2- 172.21.15.129:0/601199900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24b0195d80 0x7f24b0196230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T20:47:24.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.364+0000 7f24ae7fc700 1 --2- 172.21.15.129:0/601199900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24b019bc00 0x7f24b0195840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:47:24.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.364+0000 7f24aeffd700 1 --2- 172.21.15.129:0/601199900 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24b01066d0 0x7f24b019b6c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f24a4004f80 tx=0x7f24a4004fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:47:24.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.364+0000 7f2497fff700 1 -- 172.21.15.129:0/601199900 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4027070 con 0x7f24b01066d0 2024-03-11T20:47:24.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.364+0000 7f24b5546700 1 -- 172.21.15.129:0/601199900 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b0069ea0 con 0x7f24b01066d0 2024-03-11T20:47:24.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.364+0000 7f2497fff700 1 -- 172.21.15.129:0/601199900 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f24a4005da0 con 0x7f24b01066d0 2024-03-11T20:47:24.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.364+0000 7f2497fff700 1 -- 172.21.15.129:0/601199900 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4022c50 con 0x7f24b01066d0 2024-03-11T20:47:24.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.364+0000 7f24b5546700 1 -- 172.21.15.129:0/601199900 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b006a430 con 0x7f24b01066d0 2024-03-11T20:47:24.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.365+0000 7f2497fff700 1 -- 172.21.15.129:0/601199900 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f24a402a070 con 0x7f24b01066d0 2024-03-11T20:47:24.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.365+0000 7f2497fff700 1 --2- 172.21.15.129:0/601199900 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2498039450 0x7f249803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:24.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.365+0000 7f2497fff700 1 -- 172.21.15.129:0/601199900 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f24a404df00 con 0x7f24b01066d0 2024-03-11T20:47:24.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.365+0000 7f24ae7fc700 1 --2- 172.21.15.129:0/601199900 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2498039450 0x7f249803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:24.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.365+0000 7f24b5546700 1 -- 172.21.15.129:0/601199900 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b0101e10 con 0x7f24b01066d0 2024-03-11T20:47:24.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.366+0000 7f24ae7fc700 1 --2- 172.21.15.129:0/601199900 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2498039450 0x7f249803b910 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009ec0 tx=0x7f24a0009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:47:24.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.370+0000 7f2497fff700 1 -- 172.21.15.129:0/601199900 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f24a4013070 con 0x7f24b01066d0 2024-03-11T20:47:24.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:24 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:47:24.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:24 smithi157 ceph-mon[33359]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:24.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.578+0000 7f24b5546700 1 -- 172.21.15.129:0/601199900 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f24b0196d70 con 0x7f24b01066d0 2024-03-11T20:47:24.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.579+0000 7f2497fff700 1 -- 172.21.15.129:0/601199900 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7f24a4031070 con 0x7f24b01066d0 2024-03-11T20:47:24.575 INFO:teuthology.orchestra.run.smithi129.stdout:# minimal ceph.conf for 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:47:24.575 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-03-11T20:47:24.575 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:47:24.575 INFO:teuthology.orchestra.run.smithi129.stdout: mon_host = [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] 2024-03-11T20:47:24.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.581+0000 7f24b5546700 1 -- 172.21.15.129:0/601199900 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2498039450 msgr2=0x7f249803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:24.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.581+0000 7f24b5546700 1 --2- 172.21.15.129:0/601199900 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2498039450 0x7f249803b910 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009ec0 tx=0x7f24a0009210 comp rx=0 tx=0).stop 2024-03-11T20:47:24.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.581+0000 7f24b5546700 1 -- 172.21.15.129:0/601199900 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24b01066d0 msgr2=0x7f24b019b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:24.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.581+0000 7f24b5546700 1 --2- 172.21.15.129:0/601199900 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24b01066d0 0x7f24b019b6c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f24a4004f80 tx=0x7f24a4004fb0 comp rx=0 tx=0).stop 2024-03-11T20:47:24.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.581+0000 7f24b5546700 1 -- 172.21.15.129:0/601199900 shutdown_connections 2024-03-11T20:47:24.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.581+0000 7f24b5546700 1 --2- 172.21.15.129:0/601199900 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f2498039450 0x7f249803b910 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:24.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.581+0000 7f24b5546700 1 --2- 172.21.15.129:0/601199900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24b0195d80 0x7f24b0196230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:24.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.581+0000 7f24b5546700 1 --2- 172.21.15.129:0/601199900 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f24b01066d0 0x7f24b019b6c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:24.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.581+0000 7f24b5546700 1 --2- 172.21.15.129:0/601199900 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24b019bc00 0x7f24b0195840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:24.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.581+0000 7f24b5546700 1 -- 172.21.15.129:0/601199900 >> 172.21.15.129:0/601199900 conn(0x7f24b00fc1e0 msgr2=0x7f24b01021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:47:24.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.581+0000 7f24b5546700 1 -- 172.21.15.129:0/601199900 shutdown_connections 2024-03-11T20:47:24.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:24.581+0000 7f24b5546700 1 -- 172.21.15.129:0/601199900 wait complete. 2024-03-11T20:47:25.127 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/601199900' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:25.127 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:25.127 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:25.127 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:25.127 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:47:25.127 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:47:25.127 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:25.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/601199900' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:25.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:25.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:25.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:25.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:47:25.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:47:25.128 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:25.434 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T20:47:25.434 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:47:25.434 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T20:47:25.464 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:47:25.465 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:47:25.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/601199900' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:25.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:25.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:25.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:25.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:47:25.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:47:25.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:25.531 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-11T20:47:25.531 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T20:47:25.566 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-11T20:47:25.566 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:47:25.636 INFO:tasks.cephadm:Adding mgr.y on smithi129 2024-03-11T20:47:25.636 INFO:tasks.cephadm:Adding mgr.x on smithi157 2024-03-11T20:47:25.637 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch apply mgr '2;smithi129=y;smithi157=x' 2024-03-11T20:47:25.911 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:47:26.226 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:26 smithi129 ceph-mon[30184]: Reconfiguring mon.c (monmap changed)... 2024-03-11T20:47:26.226 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:26 smithi129 ceph-mon[30184]: Reconfiguring daemon mon.c on smithi129 2024-03-11T20:47:26.226 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:26 smithi129 ceph-mon[30184]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:26.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:26 smithi129 ceph-mon[42066]: Reconfiguring mon.c (monmap changed)... 2024-03-11T20:47:26.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:26 smithi129 ceph-mon[42066]: Reconfiguring daemon mon.c on smithi129 2024-03-11T20:47:26.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:26 smithi129 ceph-mon[42066]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:26.375 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:26 smithi157 ceph-mon[33359]: Reconfiguring mon.c (monmap changed)... 2024-03-11T20:47:26.375 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:26 smithi157 ceph-mon[33359]: Reconfiguring daemon mon.c on smithi129 2024-03-11T20:47:26.375 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:26 smithi157 ceph-mon[33359]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:27.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.575+0000 7f7eaa081700 1 -- 172.21.15.157:0/1747327774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ea4100840 msgr2=0x7f7ea4100c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:27.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.575+0000 7f7eaa081700 1 --2- 172.21.15.157:0/1747327774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ea4100840 0x7f7ea4100c60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c009910 tx=0x7f7e9c009c20 comp rx=0 tx=0).stop 2024-03-11T20:47:27.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.575+0000 7f7eaa081700 1 -- 172.21.15.157:0/1747327774 shutdown_connections 2024-03-11T20:47:27.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.576+0000 7f7eaa081700 1 --2- 172.21.15.157:0/1747327774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ea4100840 0x7f7ea4100c60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:27.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.576+0000 7f7eaa081700 1 -- 172.21.15.157:0/1747327774 >> 172.21.15.157:0/1747327774 conn(0x7f7ea40fbdc0 msgr2=0x7f7ea40fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:47:27.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.576+0000 7f7eaa081700 1 -- 172.21.15.157:0/1747327774 shutdown_connections 2024-03-11T20:47:27.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.576+0000 7f7eaa081700 1 -- 172.21.15.157:0/1747327774 wait complete. 2024-03-11T20:47:27.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.576+0000 7f7eaa081700 1 Processor -- start 2024-03-11T20:47:27.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.576+0000 7f7eaa081700 1 -- start start 2024-03-11T20:47:27.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.576+0000 7f7eaa081700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ea4100840 0x7f7ea40fdd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:27.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.576+0000 7f7eaa081700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ea40fc3d0 0x7f7ea40fc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:27.574 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7eaa081700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ea40fcd90 0x7f7ea41a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:27.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7eaa081700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea4105cf0 con 0x7f7ea40fcd90 2024-03-11T20:47:27.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7eaa081700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea4105b70 con 0x7f7ea4100840 2024-03-11T20:47:27.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7eaa081700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea4105e70 con 0x7f7ea40fc3d0 2024-03-11T20:47:27.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7ea37fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ea4100840 0x7f7ea40fdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:27.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7ea37fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ea4100840 0x7f7ea40fdd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:34054/0 (socket says 172.21.15.157:34054) 2024-03-11T20:47:27.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7ea37fe700 1 -- 172.21.15.157:0/2453262910 learned_addr learned my addr 172.21.15.157:0/2453262910 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:47:27.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7ea3fff700 1 --2- 172.21.15.157:0/2453262910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ea40fcd90 0x7f7ea41a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:27.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7ea2ffd700 1 --2- 172.21.15.157:0/2453262910 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ea40fc3d0 0x7f7ea40fc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:27.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7ea37fe700 1 -- 172.21.15.157:0/2453262910 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ea40fc3d0 msgr2=0x7f7ea40fc850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:27.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7ea37fe700 1 --2- 172.21.15.157:0/2453262910 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ea40fc3d0 0x7f7ea40fc850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:27.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7ea37fe700 1 -- 172.21.15.157:0/2453262910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ea40fcd90 msgr2=0x7f7ea41a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:27.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7ea37fe700 1 --2- 172.21.15.157:0/2453262910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ea40fcd90 0x7f7ea41a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:27.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7ea37fe700 1 -- 172.21.15.157:0/2453262910 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea41a0cf0 con 0x7f7ea4100840 2024-03-11T20:47:27.575 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7ea37fe700 1 --2- 172.21.15.157:0/2453262910 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ea4100840 0x7f7ea40fdd30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c004f90 tx=0x7f7e9c009ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:47:27.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7ea0ff9700 1 -- 172.21.15.157:0/2453262910 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c027070 con 0x7f7ea4100840 2024-03-11T20:47:27.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7ea0ff9700 1 -- 172.21.15.157:0/2453262910 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f7e9c005da0 con 0x7f7ea4100840 2024-03-11T20:47:27.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7ea0ff9700 1 -- 172.21.15.157:0/2453262910 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c022cb0 con 0x7f7ea4100840 2024-03-11T20:47:27.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7eaa081700 1 -- 172.21.15.157:0/2453262910 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea41a0f80 con 0x7f7ea4100840 2024-03-11T20:47:27.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.577+0000 7f7eaa081700 1 -- 172.21.15.157:0/2453262910 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea41a1400 con 0x7f7ea4100840 2024-03-11T20:47:27.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.578+0000 7f7ea0ff9700 1 -- 172.21.15.157:0/2453262910 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f7e9c02a070 con 0x7f7ea4100840 2024-03-11T20:47:27.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.578+0000 7f7eaa081700 1 -- 172.21.15.157:0/2453262910 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea4063410 con 0x7f7ea4100840 2024-03-11T20:47:27.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.578+0000 7f7ea0ff9700 1 --2- 172.21.15.157:0/2453262910 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f7e90039450 0x7f7e9003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:27.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.579+0000 7f7ea0ff9700 1 -- 172.21.15.157:0/2453262910 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7e9c04d310 con 0x7f7ea4100840 2024-03-11T20:47:27.576 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.579+0000 7f7ea2ffd700 1 --2- 172.21.15.157:0/2453262910 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f7e90039450 0x7f7e9003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:27.577 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.579+0000 7f7ea2ffd700 1 --2- 172.21.15.157:0/2453262910 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f7e90039450 0x7f7e9003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009ad0 tx=0x7f7e8c009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:47:27.579 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.584+0000 7f7ea0ff9700 1 -- 172.21.15.157:0/2453262910 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f7e9c013070 con 0x7f7ea4100840 2024-03-11T20:47:27.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:27 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:27.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:27 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:27.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:27 smithi157 ceph-mon[33359]: Reconfiguring mon.b (monmap changed)... 2024-03-11T20:47:27.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:27 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:47:27.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:27 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:47:27.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:27 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:27.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:27 smithi157 ceph-mon[33359]: Reconfiguring daemon mon.b on smithi157 2024-03-11T20:47:27.793 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.797+0000 7f7eaa081700 1 -- 172.21.15.157:0/2453262910 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi129=y;smithi157=x", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ea4062220 con 0x7f7e90039450 2024-03-11T20:47:27.801 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.806+0000 7f7ea0ff9700 1 -- 172.21.15.157:0/2453262910 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7ea4062220 con 0x7f7e90039450 2024-03-11T20:47:27.801 INFO:teuthology.orchestra.run.smithi157.stdout:Scheduled mgr update... 2024-03-11T20:47:27.803 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.808+0000 7f7eaa081700 1 -- 172.21.15.157:0/2453262910 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f7e90039450 msgr2=0x7f7e9003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:27.804 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.808+0000 7f7eaa081700 1 --2- 172.21.15.157:0/2453262910 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f7e90039450 0x7f7e9003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009ad0 tx=0x7f7e8c009210 comp rx=0 tx=0).stop 2024-03-11T20:47:27.804 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.808+0000 7f7eaa081700 1 -- 172.21.15.157:0/2453262910 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ea4100840 msgr2=0x7f7ea40fdd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:27.804 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.808+0000 7f7eaa081700 1 --2- 172.21.15.157:0/2453262910 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ea4100840 0x7f7ea40fdd30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c004f90 tx=0x7f7e9c009ed0 comp rx=0 tx=0).stop 2024-03-11T20:47:27.804 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.808+0000 7f7eaa081700 1 -- 172.21.15.157:0/2453262910 shutdown_connections 2024-03-11T20:47:27.804 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.808+0000 7f7eaa081700 1 --2- 172.21.15.157:0/2453262910 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f7e90039450 0x7f7e9003b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:27.804 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.808+0000 7f7eaa081700 1 --2- 172.21.15.157:0/2453262910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ea40fcd90 0x7f7ea41a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:27.804 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.808+0000 7f7eaa081700 1 --2- 172.21.15.157:0/2453262910 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ea4100840 0x7f7ea40fdd30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:27.804 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.808+0000 7f7eaa081700 1 --2- 172.21.15.157:0/2453262910 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7ea40fc3d0 0x7f7ea40fc850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:27.804 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.808+0000 7f7eaa081700 1 -- 172.21.15.157:0/2453262910 >> 172.21.15.157:0/2453262910 conn(0x7f7ea40fbdc0 msgr2=0x7f7ea406a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:47:27.804 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.808+0000 7f7eaa081700 1 -- 172.21.15.157:0/2453262910 shutdown_connections 2024-03-11T20:47:27.804 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:47:27.808+0000 7f7eaa081700 1 -- 172.21.15.157:0/2453262910 wait complete. 2024-03-11T20:47:27.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:27 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:27.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:27 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:27.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:27 smithi129 ceph-mon[30184]: Reconfiguring mon.b (monmap changed)... 2024-03-11T20:47:27.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:27 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:47:27.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:27 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:47:27.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:27 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:27.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:27 smithi129 ceph-mon[30184]: Reconfiguring daemon mon.b on smithi157 2024-03-11T20:47:27.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:27 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:27.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:27 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:27.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:27 smithi129 ceph-mon[42066]: Reconfiguring mon.b (monmap changed)... 2024-03-11T20:47:27.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:27 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T20:47:27.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:27 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T20:47:27.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:27 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:27.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:27 smithi129 ceph-mon[42066]: Reconfiguring daemon mon.b on smithi157 2024-03-11T20:47:28.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:28 smithi157 ceph-mon[33359]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:28.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:28 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:28.773 DEBUG:teuthology.orchestra.run.smithi157:mgr.x> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mgr.x.service 2024-03-11T20:47:28.776 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T20:47:28.776 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:47:28.776 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T20:47:28.794 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T20:47:28.794 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2024-03-11T20:47:28.852 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T20:47:28.852 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T20:47:28.852 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T20:47:28.852 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T20:47:28.852 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T20:47:28.852 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-11 20:47:14.744169158 +0000 2024-03-11T20:47:28.852 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-11 20:41:35.305824544 +0000 2024-03-11T20:47:28.852 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-11 20:41:35.305824544 +0000 2024-03-11T20:47:28.852 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-11 20:41:35.305824544 +0000 2024-03-11T20:47:28.853 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T20:47:28.918 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:28 smithi129 ceph-mon[30184]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:28.918 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:28 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:28.918 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:28 smithi129 ceph-mon[42066]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:28.918 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:28 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:28.921 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-11T20:47:28.921 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-11T20:47:28.921 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000228903 s, 2.2 MB/s 2024-03-11T20:47:28.922 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T20:47:28.980 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2024-03-11T20:47:29.037 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T20:47:29.037 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T20:47:29.037 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-11T20:47:29.037 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T20:47:29.037 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T20:47:29.038 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-11 20:47:14.745169136 +0000 2024-03-11T20:47:29.038 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-11 20:41:35.670816415 +0000 2024-03-11T20:47:29.038 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-11 20:41:35.670816415 +0000 2024-03-11T20:47:29.038 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-11 20:41:35.670816415 +0000 2024-03-11T20:47:29.038 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T20:47:29.105 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-11T20:47:29.105 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-11T20:47:29.105 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000226345 s, 2.3 MB/s 2024-03-11T20:47:29.107 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T20:47:29.163 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2024-03-11T20:47:29.219 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T20:47:29.219 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T20:47:29.219 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-11T20:47:29.219 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T20:47:29.219 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T20:47:29.219 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-11 20:47:14.745169136 +0000 2024-03-11T20:47:29.220 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-11 20:41:36.022808575 +0000 2024-03-11T20:47:29.220 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-11 20:41:36.022808575 +0000 2024-03-11T20:47:29.220 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-11 20:41:36.022808575 +0000 2024-03-11T20:47:29.220 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T20:47:29.287 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-11T20:47:29.287 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-11T20:47:29.287 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000210018 s, 2.4 MB/s 2024-03-11T20:47:29.288 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T20:47:29.344 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2024-03-11T20:47:29.399 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T20:47:29.399 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T20:47:29.399 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-11T20:47:29.399 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T20:47:29.399 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T20:47:29.399 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-03-11 20:47:14.746169113 +0000 2024-03-11T20:47:29.399 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-03-11 20:41:36.374800736 +0000 2024-03-11T20:47:29.400 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-03-11 20:41:36.374800736 +0000 2024-03-11T20:47:29.400 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-03-11 20:41:36.374800736 +0000 2024-03-11T20:47:29.400 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T20:47:29.466 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-03-11T20:47:29.467 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-03-11T20:47:29.467 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000212594 s, 2.4 MB/s 2024-03-11T20:47:29.467 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T20:47:29.522 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-11T20:47:29.522 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T20:47:29.537 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T20:47:29.538 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_1 2024-03-11T20:47:29.593 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T20:47:29.593 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T20:47:29.593 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-11T20:47:29.593 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T20:47:29.593 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T20:47:29.593 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-11 20:47:05.259767325 +0000 2024-03-11T20:47:29.593 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-11 20:41:35.295849001 +0000 2024-03-11T20:47:29.593 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-11 20:41:35.295849001 +0000 2024-03-11T20:47:29.593 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-11 20:41:35.295849001 +0000 2024-03-11T20:47:29.593 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T20:47:29.659 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-11T20:47:29.659 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-11T20:47:29.659 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000111585 s, 4.6 MB/s 2024-03-11T20:47:29.660 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T20:47:29.718 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_2 2024-03-11T20:47:29.774 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T20:47:29.774 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T20:47:29.774 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-11T20:47:29.774 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T20:47:29.774 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T20:47:29.774 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-11 20:47:05.259767325 +0000 2024-03-11T20:47:29.775 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-11 20:41:35.625843172 +0000 2024-03-11T20:47:29.775 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-11 20:41:35.625843172 +0000 2024-03-11T20:47:29.775 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-11 20:41:35.625843172 +0000 2024-03-11T20:47:29.775 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T20:47:29.840 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-11T20:47:29.840 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-11T20:47:29.840 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000112057 s, 4.6 MB/s 2024-03-11T20:47:29.841 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T20:47:29.868 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:29 smithi157 ceph-mon[33359]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi129=y;smithi157=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:47:29.868 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:29 smithi157 ceph-mon[33359]: Saving service mgr spec with placement smithi129=y;smithi157=x;count:2 2024-03-11T20:47:29.868 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:29.868 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:29.868 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:29.868 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:29.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:29.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:29.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T20:47:29.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T20:47:29.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T20:47:29.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:29.884 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_3 2024-03-11T20:47:29.938 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T20:47:29.939 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T20:47:29.939 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-11T20:47:29.939 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T20:47:29.939 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T20:47:29.939 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-11 20:47:05.259767325 +0000 2024-03-11T20:47:29.939 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-11 20:41:35.944837538 +0000 2024-03-11T20:47:29.939 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-11 20:41:35.944837538 +0000 2024-03-11T20:47:29.939 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-11 20:41:35.944837538 +0000 2024-03-11T20:47:29.939 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T20:47:29.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[30184]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi129=y;smithi157=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:47:29.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[30184]: Saving service mgr spec with placement smithi129=y;smithi157=x;count:2 2024-03-11T20:47:29.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:29.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:29.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:29.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:29.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:29.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:29.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T20:47:29.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T20:47:29.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T20:47:29.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:29.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[42066]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi129=y;smithi157=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:47:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[42066]: Saving service mgr spec with placement smithi129=y;smithi157=x;count:2 2024-03-11T20:47:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T20:47:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T20:47:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T20:47:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:30.021 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-11T20:47:30.021 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-11T20:47:30.021 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 9.3879e-05 s, 5.5 MB/s 2024-03-11T20:47:30.022 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T20:47:30.078 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_4 2024-03-11T20:47:30.132 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T20:47:30.133 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T20:47:30.133 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-11T20:47:30.133 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T20:47:30.133 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T20:47:30.133 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-11 20:47:05.260767307 +0000 2024-03-11T20:47:30.133 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-11 20:41:36.274831709 +0000 2024-03-11T20:47:30.133 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-11 20:41:36.274831709 +0000 2024-03-11T20:47:30.133 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-11 20:41:36.274831709 +0000 2024-03-11T20:47:30.133 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T20:47:30.199 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-11T20:47:30.199 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-11T20:47:30.199 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 9.3138e-05 s, 5.5 MB/s 2024-03-11T20:47:30.200 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T20:47:30.255 INFO:tasks.cephadm:Deploying osd.0 on smithi129 with /dev/vg_nvme/lv_4... 2024-03-11T20:47:30.255 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T20:47:30.499 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:47:30.878 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:30 smithi129 ceph-mon[30184]: Deploying daemon mgr.x on smithi157 2024-03-11T20:47:30.878 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:30 smithi129 ceph-mon[30184]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:30.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:30 smithi129 ceph-mon[42066]: Deploying daemon mgr.x on smithi157 2024-03-11T20:47:30.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:30 smithi129 ceph-mon[42066]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:30.882 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:30 smithi157 ceph-mon[33359]: Deploying daemon mgr.x on smithi157 2024-03-11T20:47:30.882 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:30 smithi157 ceph-mon[33359]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:31.485 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:31 smithi157 systemd[1]: Starting Ceph mgr.x for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T20:47:31.799 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:31 smithi157 podman[35442]: 2024-03-11 20:47:31.43963947 +0000 UTC m=+0.017818078 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T20:47:31.799 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:31 smithi157 podman[35442]: 2024-03-11 20:47:31.539839399 +0000 UTC m=+0.118017999 container create 41d4789ab85b8de7f19c53790719266841b683462f1acae8292cb1cab42ca198 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-2b0101e) 2024-03-11T20:47:31.800 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:31 smithi157 podman[35442]: 2024-03-11 20:47:31.775472747 +0000 UTC m=+0.353651358 container init 41d4789ab85b8de7f19c53790719266841b683462f1acae8292cb1cab42ca198 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-11T20:47:31.800 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:31 smithi157 podman[35442]: 2024-03-11 20:47:31.780592189 +0000 UTC m=+0.358770790 container start 41d4789ab85b8de7f19c53790719266841b683462f1acae8292cb1cab42ca198 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-11T20:47:32.235 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:31 smithi157 ceph-mgr[35480]: -- 172.21.15.157:0/4130662953 <== mon.1 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556207f7c5a0 con 0x556207f50c00 2024-03-11T20:47:32.235 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:31 smithi157 bash[35442]: 41d4789ab85b8de7f19c53790719266841b683462f1acae8292cb1cab42ca198 2024-03-11T20:47:32.235 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:31 smithi157 systemd[1]: Started Ceph mgr.x for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T20:47:32.597 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T20:47:32.598 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T20:47:32.598 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-03-11T20:47:32.598 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-03-11T20:47:32.598 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113765 s, 922 MB/s 2024-03-11T20:47:32.598 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-03-11T20:47:32.606 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:32 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:32.343+0000 7f86ce9f6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T20:47:32.606 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:32 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:32.441+0000 7f86ce9f6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T20:47:32.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:32 smithi129 ceph-mon[30184]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:32.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:32 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:32.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:32 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:32.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:32 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:32.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:32 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:32.913 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:32 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:32.913 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:32 smithi129 ceph-mon[42066]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:32.913 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:32 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:32.913 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:32 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:32.914 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:32 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:32.914 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:32 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:32.914 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:32 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:32.955 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:32 smithi157 ceph-mon[33359]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:32.955 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:32 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:32.956 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:32 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:32.956 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:32 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:32.956 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:32 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:32.956 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:32 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:33.215 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi129:vg_nvme/lv_4 2024-03-11T20:47:33.234 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:33.144+0000 7f86ce9f6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T20:47:33.234 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:33.239+0000 7f86ce9f6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T20:47:33.451 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:47:33.484 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:33.383+0000 7f86ce9f6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T20:47:33.484 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:33.486+0000 7f86ce9f6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T20:47:33.844 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:33.573+0000 7f86ce9f6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T20:47:33.844 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:33.848+0000 7f86ce9f6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T20:47:34.135 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:33.957+0000 7f86ce9f6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T20:47:34.135 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:34.044+0000 7f86ce9f6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T20:47:34.135 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:34.139+0000 7f86ce9f6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T20:47:34.484 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:34.329+0000 7f86ce9f6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T20:47:34.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:34.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:34.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:34.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:34.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:34.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[30184]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T20:47:34.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T20:47:34.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T20:47:34.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:34.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[30184]: Reconfiguring daemon mgr.y on smithi129 2024-03-11T20:47:34.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[30184]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:34.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:34.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:34.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:34.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:34.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:34.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[42066]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T20:47:34.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T20:47:34.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T20:47:34.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:34.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[42066]: Reconfiguring daemon mgr.y on smithi129 2024-03-11T20:47:34.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:34 smithi129 ceph-mon[42066]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:34.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:34 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:34.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:34 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:34.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:34 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:34.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:34 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:34.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:34 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:34.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:34 smithi157 ceph-mon[33359]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T20:47:34.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:34 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T20:47:34.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:34 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T20:47:34.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:34 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:34.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:34 smithi157 ceph-mon[33359]: Reconfiguring daemon mgr.y on smithi129 2024-03-11T20:47:34.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:34 smithi157 ceph-mon[33359]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:35.635 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:35 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:35.311+0000 7f86ce9f6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T20:47:35.635 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:35 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:35.443+0000 7f86ce9f6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T20:47:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.794+0000 7fec056c6700 1 -- 172.21.15.129:0/2667041381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec00100d10 msgr2=0x7fec00101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.794+0000 7fec056c6700 1 --2- 172.21.15.129:0/2667041381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec00100d10 0x7fec00101130 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7febf00099c0 tx=0x7febf0009cd0 comp rx=0 tx=0).stop 2024-03-11T20:47:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.794+0000 7fec056c6700 1 -- 172.21.15.129:0/2667041381 shutdown_connections 2024-03-11T20:47:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.794+0000 7fec056c6700 1 --2- 172.21.15.129:0/2667041381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fec001028b0 0x7fec00109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.794+0000 7fec056c6700 1 --2- 172.21.15.129:0/2667041381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec00100d10 0x7fec00101130 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.794+0000 7fec056c6700 1 --2- 172.21.15.129:0/2667041381 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec00101f10 0x7fec00102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.794+0000 7fec056c6700 1 -- 172.21.15.129:0/2667041381 >> 172.21.15.129:0/2667041381 conn(0x7fec000fc2b0 msgr2=0x7fec000fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:47:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.794+0000 7fec056c6700 1 -- 172.21.15.129:0/2667041381 shutdown_connections 2024-03-11T20:47:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.794+0000 7fec056c6700 1 -- 172.21.15.129:0/2667041381 wait complete. 2024-03-11T20:47:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.795+0000 7fec056c6700 1 Processor -- start 2024-03-11T20:47:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.795+0000 7fec056c6700 1 -- start start 2024-03-11T20:47:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.795+0000 7fec056c6700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fec00100d10 0x7fec00196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.795+0000 7fec056c6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec00101f10 0x7fec00196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.795+0000 7fec056c6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec001028b0 0x7fec0019dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.795+0000 7fec056c6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0010bdb0 con 0x7fec00101f10 2024-03-11T20:47:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.795+0000 7fec056c6700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0010bc30 con 0x7fec00100d10 2024-03-11T20:47:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.795+0000 7fec056c6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0010bf30 con 0x7fec001028b0 2024-03-11T20:47:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.795+0000 7febff7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec001028b0 0x7fec0019dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.795+0000 7febfe7fc700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec00101f10 0x7fec00196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.795+0000 7febff7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec001028b0 0x7fec0019dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:52340/0 (socket says 172.21.15.129:52340) 2024-03-11T20:47:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.795+0000 7febfeffd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fec00100d10 0x7fec00196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.795+0000 7febff7fe700 1 -- 172.21.15.129:0/3573222823 learned_addr learned my addr 172.21.15.129:0/3573222823 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:47:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.795+0000 7febfe7fc700 1 -- 172.21.15.129:0/3573222823 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec001028b0 msgr2=0x7fec0019dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.795+0000 7febfe7fc700 1 --2- 172.21.15.129:0/3573222823 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec001028b0 0x7fec0019dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.796+0000 7febfe7fc700 1 -- 172.21.15.129:0/3573222823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fec00100d10 msgr2=0x7fec00196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.796+0000 7febfe7fc700 1 --2- 172.21.15.129:0/3573222823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fec00100d10 0x7fec00196520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.796+0000 7febfe7fc700 1 -- 172.21.15.129:0/3573222823 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0019e180 con 0x7fec00101f10 2024-03-11T20:47:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.796+0000 7febff7fe700 1 --2- 172.21.15.129:0/3573222823 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec001028b0 0x7fec0019dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:47:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.796+0000 7febfe7fc700 1 --2- 172.21.15.129:0/3573222823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec00101f10 0x7fec00196a60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7febf400ea20 tx=0x7febf400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:47:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.796+0000 7febe7fff700 1 -- 172.21.15.129:0/3573222823 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf4004220 con 0x7fec00101f10 2024-03-11T20:47:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.796+0000 7febe7fff700 1 -- 172.21.15.129:0/3573222823 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7febf400cdf0 con 0x7fec00101f10 2024-03-11T20:47:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.796+0000 7fec056c6700 1 -- 172.21.15.129:0/3573222823 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec00069af0 con 0x7fec00101f10 2024-03-11T20:47:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.796+0000 7febe7fff700 1 -- 172.21.15.129:0/3573222823 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf4003c10 con 0x7fec00101f10 2024-03-11T20:47:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.796+0000 7fec056c6700 1 -- 172.21.15.129:0/3573222823 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec0006a030 con 0x7fec00101f10 2024-03-11T20:47:35.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.797+0000 7febe7fff700 1 -- 172.21.15.129:0/3573222823 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7febf4004830 con 0x7fec00101f10 2024-03-11T20:47:35.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.797+0000 7fec056c6700 1 -- 172.21.15.129:0/3573222823 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec00068aa0 con 0x7fec00101f10 2024-03-11T20:47:35.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.797+0000 7febe7fff700 1 --2- 172.21.15.129:0/3573222823 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7febe8039200 0x7febe803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:47:35.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.797+0000 7febe7fff700 1 -- 172.21.15.129:0/3573222823 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7febf404c2c0 con 0x7fec00101f10 2024-03-11T20:47:35.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.797+0000 7febfeffd700 1 --2- 172.21.15.129:0/3573222823 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7febe8039200 0x7febe803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:47:35.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.797+0000 7febfeffd700 1 --2- 172.21.15.129:0/3573222823 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7febe8039200 0x7febe803b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7febf0006010 tx=0x7febf0005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:47:35.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:35.802+0000 7febe7fff700 1 -- 172.21.15.129:0/3573222823 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7febf4020390 con 0x7fec00101f10 2024-03-11T20:47:35.903 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:35 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:35.903 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:35 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:35.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:35 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:35.985 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:35 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:35.778+0000 7f86ce9f6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T20:47:35.985 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:35 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:35.868+0000 7f86ce9f6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T20:47:36.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:36.014+0000 7fec056c6700 1 -- 172.21.15.129:0/3573222823 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fec0006a350 con 0x7febe8039200 2024-03-11T20:47:36.485 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:36 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:36.064+0000 7f86ce9f6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:47:36.485 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:36 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:36.247+0000 7f86ce9f6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T20:47:36.791 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:36 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:36.534+0000 7f86ce9f6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:47:36.792 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:36 smithi157 ceph-mon[33359]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:36.792 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:47:36.792 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:47:36.792 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:36.792 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:36.792 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:36.792 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:36.792 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:36.792 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:36.792 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:36.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[42066]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:36.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:47:36.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:47:36.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:36.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:36.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:36.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:36.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:36.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:36.878 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:36.879 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[30184]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:36.879 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:47:36.879 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:47:36.879 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:36.879 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:36.879 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:36.879 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:36.879 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:36.879 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:47:36.879 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:37.235 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:36 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:36.796+0000 7f86ce9f6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T20:47:37.235 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:37 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:37.029+0000 7f86ce9f6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T20:47:37.853 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:37 smithi129 ceph-mon[30184]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:47:37.853 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:37 smithi129 ceph-mon[42066]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:47:37.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:37 smithi157 ceph-mon[33359]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:47:37.985 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:37 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:37.836+0000 7f86ce9f6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T20:47:38.720 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:38 smithi157 ceph-mon[33359]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:38.720 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:38 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:38.410+0000 7f86ce9f6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T20:47:38.720 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:38 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:38.509+0000 7f86ce9f6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T20:47:38.720 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:47:38 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:47:38.725+0000 7f86ce9f6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T20:47:38.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:38 smithi129 ceph-mon[30184]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:38.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:38 smithi129 ceph-mon[42066]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:39.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:39.714+0000 7febe7fff700 1 -- 172.21.15.129:0/3573222823 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mgrmap(e 14) v1 ==== 91699+0+0 (secure 0 0 0) 0x7febf4005540 con 0x7fec00101f10 2024-03-11T20:47:39.905 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3584699068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5dbe3ee2-8c30-4242-baf0-a36e1ac0be8f"}]: dispatch 2024-03-11T20:47:39.906 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5dbe3ee2-8c30-4242-baf0-a36e1ac0be8f"}]: dispatch 2024-03-11T20:47:39.906 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5dbe3ee2-8c30-4242-baf0-a36e1ac0be8f"}]': finished 2024-03-11T20:47:39.906 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:39 smithi157 ceph-mon[33359]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T20:47:39.906 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:39 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:39.906 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:39 smithi157 ceph-mon[33359]: Standby manager daemon x started 2024-03-11T20:47:39.906 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:39 smithi157 ceph-mon[33359]: from='mgr.? 172.21.15.157:0/2462245836' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T20:47:39.906 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:39 smithi157 ceph-mon[33359]: from='mgr.? 172.21.15.157:0/2462245836' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T20:47:39.906 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:39 smithi157 ceph-mon[33359]: from='mgr.? 172.21.15.157:0/2462245836' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T20:47:39.906 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:39 smithi157 ceph-mon[33359]: from='mgr.? 172.21.15.157:0/2462245836' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T20:47:39.906 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/180803079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:47:39.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3584699068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5dbe3ee2-8c30-4242-baf0-a36e1ac0be8f"}]: dispatch 2024-03-11T20:47:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5dbe3ee2-8c30-4242-baf0-a36e1ac0be8f"}]: dispatch 2024-03-11T20:47:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5dbe3ee2-8c30-4242-baf0-a36e1ac0be8f"}]': finished 2024-03-11T20:47:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[30184]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T20:47:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[30184]: Standby manager daemon x started 2024-03-11T20:47:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[30184]: from='mgr.? 172.21.15.157:0/2462245836' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T20:47:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[30184]: from='mgr.? 172.21.15.157:0/2462245836' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T20:47:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[30184]: from='mgr.? 172.21.15.157:0/2462245836' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T20:47:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[30184]: from='mgr.? 172.21.15.157:0/2462245836' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T20:47:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/180803079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:47:39.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3584699068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5dbe3ee2-8c30-4242-baf0-a36e1ac0be8f"}]: dispatch 2024-03-11T20:47:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5dbe3ee2-8c30-4242-baf0-a36e1ac0be8f"}]: dispatch 2024-03-11T20:47:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5dbe3ee2-8c30-4242-baf0-a36e1ac0be8f"}]': finished 2024-03-11T20:47:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[42066]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T20:47:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[42066]: Standby manager daemon x started 2024-03-11T20:47:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[42066]: from='mgr.? 172.21.15.157:0/2462245836' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T20:47:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[42066]: from='mgr.? 172.21.15.157:0/2462245836' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T20:47:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[42066]: from='mgr.? 172.21.15.157:0/2462245836' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T20:47:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[42066]: from='mgr.? 172.21.15.157:0/2462245836' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T20:47:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/180803079' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:47:40.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:40 smithi129 ceph-mon[42066]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:40.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:40 smithi129 ceph-mon[42066]: mgrmap e14: y(active, since 70s), standbys: x 2024-03-11T20:47:40.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:40 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T20:47:40.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:40 smithi129 ceph-mon[30184]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:40.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:40 smithi129 ceph-mon[30184]: mgrmap e14: y(active, since 70s), standbys: x 2024-03-11T20:47:40.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:40 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T20:47:40.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:40 smithi157 ceph-mon[33359]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:40.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:40 smithi157 ceph-mon[33359]: mgrmap e14: y(active, since 70s), standbys: x 2024-03-11T20:47:40.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:40 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T20:47:42.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:42 smithi129 ceph-mon[30184]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:42.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:42 smithi129 ceph-mon[42066]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:42.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:42 smithi157 ceph-mon[33359]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:44.954 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:44 smithi129 ceph-mon[30184]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:44.954 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:44 smithi129 ceph-mon[42066]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:44.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:44 smithi157 ceph-mon[33359]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:45.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:45 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T20:47:45.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:45 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:45.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:45 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T20:47:45.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:45 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:45.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:45 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T20:47:45.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:45 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:47:46.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:46 smithi129 ceph-mon[30184]: Deploying daemon osd.0 on smithi129 2024-03-11T20:47:46.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:46 smithi129 ceph-mon[30184]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:46.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:46 smithi129 ceph-mon[42066]: Deploying daemon osd.0 on smithi129 2024-03-11T20:47:46.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:46 smithi129 ceph-mon[42066]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:46.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:46 smithi157 ceph-mon[33359]: Deploying daemon osd.0 on smithi129 2024-03-11T20:47:46.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:46 smithi157 ceph-mon[33359]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:48.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:48 smithi129 ceph-mon[30184]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:48.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:48 smithi129 ceph-mon[42066]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:48.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:48 smithi157 ceph-mon[33359]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:50.752 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:50 smithi129 ceph-mon[30184]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:50.752 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:50 smithi129 ceph-mon[42066]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:50.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:50 smithi157 ceph-mon[33359]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:51.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:51 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:51.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:51 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:51.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:51 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:51.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:51 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:51.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:51 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:51.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:51 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:51.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:51 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:47:51.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:51 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:51.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:51 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:52.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:52 smithi157 ceph-mon[33359]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:53.053 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:52 smithi129 ceph-mon[30184]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:53.053 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:52 smithi129 ceph-mon[42066]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:54.311 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 0 on host 'smithi129' 2024-03-11T20:47:54.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:54.316+0000 7febe7fff700 1 -- 172.21.15.129:0/3573222823 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fec0006a350 con 0x7febe8039200 2024-03-11T20:47:54.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:54.318+0000 7fec056c6700 1 -- 172.21.15.129:0/3573222823 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7febe8039200 msgr2=0x7febe803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:54.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:54.318+0000 7fec056c6700 1 --2- 172.21.15.129:0/3573222823 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7febe8039200 0x7febe803b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7febf0006010 tx=0x7febf0005c60 comp rx=0 tx=0).stop 2024-03-11T20:47:54.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:54.318+0000 7fec056c6700 1 -- 172.21.15.129:0/3573222823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec00101f10 msgr2=0x7fec00196a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:47:54.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:54.318+0000 7fec056c6700 1 --2- 172.21.15.129:0/3573222823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec00101f10 0x7fec00196a60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7febf400ea20 tx=0x7febf400ed30 comp rx=0 tx=0).stop 2024-03-11T20:47:54.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:54.318+0000 7fec056c6700 1 -- 172.21.15.129:0/3573222823 shutdown_connections 2024-03-11T20:47:54.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:54.318+0000 7fec056c6700 1 --2- 172.21.15.129:0/3573222823 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7febe8039200 0x7febe803b6c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:54.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:54.318+0000 7fec056c6700 1 --2- 172.21.15.129:0/3573222823 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fec001028b0 0x7fec0019dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:54.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:54.318+0000 7fec056c6700 1 --2- 172.21.15.129:0/3573222823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fec00100d10 0x7fec00196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:54.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:54.318+0000 7fec056c6700 1 --2- 172.21.15.129:0/3573222823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec00101f10 0x7fec00196a60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:47:54.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:54.318+0000 7fec056c6700 1 -- 172.21.15.129:0/3573222823 >> 172.21.15.129:0/3573222823 conn(0x7fec000fc2b0 msgr2=0x7fec000fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:47:54.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:54.319+0000 7fec056c6700 1 -- 172.21.15.129:0/3573222823 shutdown_connections 2024-03-11T20:47:54.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:47:54.319+0000 7fec056c6700 1 -- 172.21.15.129:0/3573222823 wait complete. 2024-03-11T20:47:54.955 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:54 smithi129 ceph-mon[42066]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:54.955 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:54 smithi129 ceph-mon[42066]: from='osd.0 [v2:172.21.15.129:6802/3618360833,v1:172.21.15.129:6803/3618360833]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T20:47:54.955 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:54 smithi129 ceph-mon[42066]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T20:47:54.955 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:54.955 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:54.956 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:54 smithi129 ceph-mon[30184]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:54.956 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:54 smithi129 ceph-mon[30184]: from='osd.0 [v2:172.21.15.129:6802/3618360833,v1:172.21.15.129:6803/3618360833]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T20:47:54.956 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:54 smithi129 ceph-mon[30184]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T20:47:54.956 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:54.956 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:54.990 DEBUG:teuthology.orchestra.run.smithi129:osd.0> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T20:47:54.992 INFO:tasks.cephadm:Deploying osd.1 on smithi129 with /dev/vg_nvme/lv_3... 2024-03-11T20:47:54.992 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T20:47:55.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:54 smithi157 ceph-mon[33359]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:55.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:54 smithi157 ceph-mon[33359]: from='osd.0 [v2:172.21.15.129:6802/3618360833,v1:172.21.15.129:6803/3618360833]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T20:47:55.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:54 smithi157 ceph-mon[33359]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T20:47:55.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:55.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:55.273 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:47:55.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[30184]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T20:47:55.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[30184]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T20:47:55.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[30184]: from='osd.0 [v2:172.21.15.129:6802/3618360833,v1:172.21.15.129:6803/3618360833]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:47:55.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:55.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[30184]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:47:55.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:55.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:55.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[30184]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-11T20:47:55.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[30184]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T20:47:55.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:55.979 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-0[46443]: 2024-03-11T20:47:55.742+0000 7f002891b700 -1 osd.0 0 waiting for initial osdmap 2024-03-11T20:47:55.979 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-0[46443]: 2024-03-11T20:47:55.746+0000 7f002470e700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T20:47:55.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[42066]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T20:47:55.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[42066]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T20:47:55.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[42066]: from='osd.0 [v2:172.21.15.129:6802/3618360833,v1:172.21.15.129:6803/3618360833]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:47:55.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:55.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[42066]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:47:55.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:55.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:55.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[42066]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-11T20:47:55.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[42066]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T20:47:55.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:55 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:56.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:55 smithi157 ceph-mon[33359]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T20:47:56.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:55 smithi157 ceph-mon[33359]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T20:47:56.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:55 smithi157 ceph-mon[33359]: from='osd.0 [v2:172.21.15.129:6802/3618360833,v1:172.21.15.129:6803/3618360833]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:47:56.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:55 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:56.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:55 smithi157 ceph-mon[33359]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:47:56.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:55 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:56.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:55 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:47:56.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:55 smithi157 ceph-mon[33359]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-11T20:47:56.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:55 smithi157 ceph-mon[33359]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T20:47:56.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:55 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:56.871 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:56 smithi129 ceph-mon[30184]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:56.871 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:56 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:56.872 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:56 smithi129 ceph-mon[42066]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:56.872 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:56 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:57.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:56 smithi157 ceph-mon[33359]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:57.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:56 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:57.887 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T20:47:57.887 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T20:47:57.887 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-03-11T20:47:57.887 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-03-11T20:47:57.887 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120036 s, 874 MB/s 2024-03-11T20:47:57.887 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-03-11T20:47:58.227 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:57 smithi129 ceph-mon[30184]: purged_snaps scrub starts 2024-03-11T20:47:58.227 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:57 smithi129 ceph-mon[30184]: purged_snaps scrub ok 2024-03-11T20:47:58.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:57 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:58.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:57 smithi129 ceph-mon[30184]: osd.0 [v2:172.21.15.129:6802/3618360833,v1:172.21.15.129:6803/3618360833] boot 2024-03-11T20:47:58.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:57 smithi129 ceph-mon[30184]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T20:47:58.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:57 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:58.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:57 smithi129 ceph-mon[30184]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:58.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:57 smithi129 ceph-mon[42066]: purged_snaps scrub starts 2024-03-11T20:47:58.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:57 smithi129 ceph-mon[42066]: purged_snaps scrub ok 2024-03-11T20:47:58.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:57 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:58.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:57 smithi129 ceph-mon[42066]: osd.0 [v2:172.21.15.129:6802/3618360833,v1:172.21.15.129:6803/3618360833] boot 2024-03-11T20:47:58.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:57 smithi129 ceph-mon[42066]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T20:47:58.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:57 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:58.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:57 smithi129 ceph-mon[42066]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:58.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:57 smithi157 ceph-mon[33359]: purged_snaps scrub starts 2024-03-11T20:47:58.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:57 smithi157 ceph-mon[33359]: purged_snaps scrub ok 2024-03-11T20:47:58.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:57 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:58.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:57 smithi157 ceph-mon[33359]: osd.0 [v2:172.21.15.129:6802/3618360833,v1:172.21.15.129:6803/3618360833] boot 2024-03-11T20:47:58.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:57 smithi157 ceph-mon[33359]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T20:47:58.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:57 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:47:58.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:57 smithi157 ceph-mon[33359]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T20:47:58.796 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:58 smithi129 ceph-mon[42066]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T20:47:58.796 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:58 smithi129 ceph-mon[30184]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T20:47:59.114 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi129:vg_nvme/lv_3 2024-03-11T20:47:59.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:58 smithi157 ceph-mon[33359]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T20:47:59.850 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:47:59 smithi129 ceph-mon[30184]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:47:59.850 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:47:59 smithi129 ceph-mon[42066]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:47:59.888 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:48:00.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:47:59 smithi157 ceph-mon[33359]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:01.031 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[42066]: Detected new or changed devices on smithi129 2024-03-11T20:48:01.031 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:01.031 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:01.031 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:01.031 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[42066]: Adjusting osd_memory_target on smithi129 to 16064M 2024-03-11T20:48:01.031 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:01.032 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:01.032 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:01.032 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:01.032 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:01.032 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:01.032 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:01.032 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:01.032 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[30184]: Detected new or changed devices on smithi129 2024-03-11T20:48:01.032 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:01.032 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:01.032 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:01.032 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[30184]: Adjusting osd_memory_target on smithi129 to 16064M 2024-03-11T20:48:01.032 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:01.033 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:01.033 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:01.033 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:01.033 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:01.033 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:01.033 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:01.033 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:00 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:01.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:00 smithi157 ceph-mon[33359]: Detected new or changed devices on smithi129 2024-03-11T20:48:01.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:00 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:01.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:00 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:01.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:00 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:01.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:00 smithi157 ceph-mon[33359]: Adjusting osd_memory_target on smithi129 to 16064M 2024-03-11T20:48:01.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:00 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:01.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:00 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:01.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:00 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:01.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:00 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:01.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:00 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:01.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:00 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:01.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:00 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:01.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:00 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:01.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.538+0000 7f4eec875700 1 -- 172.21.15.129:0/1696436481 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ee8100d10 msgr2=0x7f4ee8101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:48:01.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.538+0000 7f4eec875700 1 --2- 172.21.15.129:0/1696436481 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ee8100d10 0x7f4ee8101130 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4edc0098f0 tx=0x7f4edc009c00 comp rx=0 tx=0).stop 2024-03-11T20:48:01.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.538+0000 7f4eec875700 1 -- 172.21.15.129:0/1696436481 shutdown_connections 2024-03-11T20:48:01.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.538+0000 7f4eec875700 1 --2- 172.21.15.129:0/1696436481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ee81028b0 0x7f4ee8109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:01.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.538+0000 7f4eec875700 1 --2- 172.21.15.129:0/1696436481 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ee8100d10 0x7f4ee8101130 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:01.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.538+0000 7f4eec875700 1 --2- 172.21.15.129:0/1696436481 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ee8101f10 0x7f4ee8102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:01.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.538+0000 7f4eec875700 1 -- 172.21.15.129:0/1696436481 >> 172.21.15.129:0/1696436481 conn(0x7f4ee80fc2b0 msgr2=0x7f4ee80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:48:01.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.538+0000 7f4eec875700 1 -- 172.21.15.129:0/1696436481 shutdown_connections 2024-03-11T20:48:01.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.538+0000 7f4eec875700 1 -- 172.21.15.129:0/1696436481 wait complete. 2024-03-11T20:48:01.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.539+0000 7f4eec875700 1 Processor -- start 2024-03-11T20:48:01.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.539+0000 7f4eec875700 1 -- start start 2024-03-11T20:48:01.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.539+0000 7f4eec875700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ee8100d10 0x7f4ee81976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:48:01.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.539+0000 7f4eec875700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ee8101f10 0x7f4ee8197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:48:01.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.539+0000 7f4eec875700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ee81028b0 0x7f4ee819ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:48:01.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.539+0000 7f4eec875700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee810bdb0 con 0x7f4ee8100d10 2024-03-11T20:48:01.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.539+0000 7f4eec875700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee810bc30 con 0x7f4ee8101f10 2024-03-11T20:48:01.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.539+0000 7f4eec875700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee810bf30 con 0x7f4ee81028b0 2024-03-11T20:48:01.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.539+0000 7f4ee659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ee8100d10 0x7f4ee81976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:48:01.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.539+0000 7f4ee6d9d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ee81028b0 0x7f4ee819ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:48:01.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.539+0000 7f4ee659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ee8100d10 0x7f4ee81976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44864/0 (socket says 172.21.15.129:44864) 2024-03-11T20:48:01.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.539+0000 7f4ee659c700 1 -- 172.21.15.129:0/19253201 learned_addr learned my addr 172.21.15.129:0/19253201 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:48:01.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.539+0000 7f4ee5d9b700 1 --2- 172.21.15.129:0/19253201 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ee8101f10 0x7f4ee8197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:48:01.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.539+0000 7f4ee659c700 1 -- 172.21.15.129:0/19253201 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ee81028b0 msgr2=0x7f4ee819ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:48:01.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.540+0000 7f4ee659c700 1 --2- 172.21.15.129:0/19253201 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ee81028b0 0x7f4ee819ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:01.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.540+0000 7f4ee659c700 1 -- 172.21.15.129:0/19253201 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ee8101f10 msgr2=0x7f4ee8197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:48:01.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.540+0000 7f4ee659c700 1 --2- 172.21.15.129:0/19253201 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ee8101f10 0x7f4ee8197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:01.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.540+0000 7f4ee659c700 1 -- 172.21.15.129:0/19253201 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee819f2d0 con 0x7f4ee8100d10 2024-03-11T20:48:01.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.540+0000 7f4ee6d9d700 1 --2- 172.21.15.129:0/19253201 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ee81028b0 0x7f4ee819ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:48:01.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.540+0000 7f4ee659c700 1 --2- 172.21.15.129:0/19253201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ee8100d10 0x7f4ee81976c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00f980 tx=0x7f4edc00f9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:48:01.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.540+0000 7f4ed77fe700 1 -- 172.21.15.129:0/19253201 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc00b920 con 0x7f4ee8100d10 2024-03-11T20:48:01.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.540+0000 7f4ed77fe700 1 -- 172.21.15.129:0/19253201 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4edc00bac0 con 0x7f4ee8100d10 2024-03-11T20:48:01.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.540+0000 7f4eec875700 1 -- 172.21.15.129:0/19253201 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee819f560 con 0x7f4ee8100d10 2024-03-11T20:48:01.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.540+0000 7f4ed77fe700 1 -- 172.21.15.129:0/19253201 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc004c50 con 0x7f4ee8100d10 2024-03-11T20:48:01.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.540+0000 7f4eec875700 1 -- 172.21.15.129:0/19253201 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee819fa10 con 0x7f4ee8100d10 2024-03-11T20:48:01.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.541+0000 7f4eec875700 1 -- 172.21.15.129:0/19253201 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee8068aa0 con 0x7f4ee8100d10 2024-03-11T20:48:01.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.541+0000 7f4ed77fe700 1 -- 172.21.15.129:0/19253201 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 14) v1 ==== 91699+0+0 (secure 0 0 0) 0x7f4edc02a440 con 0x7f4ee8100d10 2024-03-11T20:48:01.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.542+0000 7f4ed77fe700 1 --2- 172.21.15.129:0/19253201 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4ed006dd00 0x7f4ed00701c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:48:01.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.542+0000 7f4ed77fe700 1 -- 172.21.15.129:0/19253201 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f4edc08d7a0 con 0x7f4ee8100d10 2024-03-11T20:48:01.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.542+0000 7f4ee5d9b700 1 --2- 172.21.15.129:0/19253201 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4ed006dd00 0x7f4ed00701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:48:01.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.542+0000 7f4ee5d9b700 1 --2- 172.21.15.129:0/19253201 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4ed006dd00 0x7f4ed00701c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009750 tx=0x7f4ed8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:48:01.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.547+0000 7f4ed77fe700 1 -- 172.21.15.129:0/19253201 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f4edc013030 con 0x7f4ee8100d10 2024-03-11T20:48:01.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:01.751+0000 7f4eec875700 1 -- 172.21.15.129:0/19253201 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ee802d970 con 0x7f4ed006dd00 2024-03-11T20:48:02.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:01 smithi129 ceph-mon[42066]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:02.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:01 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:48:02.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:01 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:48:02.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:01 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:02.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:01 smithi129 ceph-mon[30184]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:02.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:01 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:48:02.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:01 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:48:02.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:01 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:02.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:01 smithi157 ceph-mon[33359]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:02.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:01 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:48:02.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:01 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:48:02.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:01 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:02.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:02 smithi129 ceph-mon[30184]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:48:02.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:02 smithi129 ceph-mon[42066]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:48:03.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:02 smithi157 ceph-mon[33359]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:48:04.227 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:03 smithi129 ceph-mon[30184]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:04.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:03 smithi129 ceph-mon[42066]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:04.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:03 smithi157 ceph-mon[33359]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:05.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3647004867' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5495628-e325-4a58-b682-50a26ef780fb"}]: dispatch 2024-03-11T20:48:05.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3647004867' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5495628-e325-4a58-b682-50a26ef780fb"}]': finished 2024-03-11T20:48:05.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:04 smithi129 ceph-mon[30184]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T20:48:05.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:04 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:05.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3647004867' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5495628-e325-4a58-b682-50a26ef780fb"}]: dispatch 2024-03-11T20:48:05.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3647004867' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5495628-e325-4a58-b682-50a26ef780fb"}]': finished 2024-03-11T20:48:05.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:04 smithi129 ceph-mon[42066]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T20:48:05.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:04 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:05.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3647004867' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5495628-e325-4a58-b682-50a26ef780fb"}]: dispatch 2024-03-11T20:48:05.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3647004867' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5495628-e325-4a58-b682-50a26ef780fb"}]': finished 2024-03-11T20:48:05.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:04 smithi157 ceph-mon[33359]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T20:48:05.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:04 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:06.227 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1438557354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:48:06.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:05 smithi129 ceph-mon[30184]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:06.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1438557354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:48:06.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:05 smithi129 ceph-mon[42066]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:06.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1438557354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:48:06.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:05 smithi157 ceph-mon[33359]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:08.834 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:08 smithi129 ceph-mon[30184]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:08.835 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:08 smithi129 ceph-mon[42066]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:08.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:08 smithi157 ceph-mon[33359]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:10.910 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:10 smithi129 ceph-mon[30184]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:10.910 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:10 smithi129 ceph-mon[42066]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:10.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:10 smithi157 ceph-mon[33359]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:11.904 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T20:48:11.904 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:11.905 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:11 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T20:48:11.905 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:11 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:11.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:11 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T20:48:11.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:11 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:12.950 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:12 smithi129 ceph-mon[30184]: Deploying daemon osd.1 on smithi129 2024-03-11T20:48:12.950 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:12 smithi129 ceph-mon[30184]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:12.951 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:12 smithi129 ceph-mon[42066]: Deploying daemon osd.1 on smithi129 2024-03-11T20:48:12.951 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:12 smithi129 ceph-mon[42066]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:12.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:12 smithi157 ceph-mon[33359]: Deploying daemon osd.1 on smithi129 2024-03-11T20:48:12.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:12 smithi157 ceph-mon[33359]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:14.976 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:14 smithi129 ceph-mon[30184]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:14.976 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:14 smithi129 ceph-mon[42066]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:14.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:14 smithi157 ceph-mon[33359]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:16.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:16 smithi129 ceph-mon[30184]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:16.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:16 smithi129 ceph-mon[42066]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:16.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:16 smithi157 ceph-mon[33359]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:17.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:17 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:17.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:17 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:17.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:17 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:17.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:17 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:17.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:17 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:17.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:17 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:17.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:17 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:17.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:17 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:17.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:17 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:18.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:18 smithi129 ceph-mon[30184]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:18.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:18 smithi129 ceph-mon[42066]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:18.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:18 smithi157 ceph-mon[33359]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:20.149 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 1 on host 'smithi129' 2024-03-11T20:48:20.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:20.154+0000 7f4ed77fe700 1 -- 172.21.15.129:0/19253201 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4ee802d970 con 0x7f4ed006dd00 2024-03-11T20:48:20.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:20.157+0000 7f4eec875700 1 -- 172.21.15.129:0/19253201 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4ed006dd00 msgr2=0x7f4ed00701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:48:20.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:20.157+0000 7f4eec875700 1 --2- 172.21.15.129:0/19253201 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4ed006dd00 0x7f4ed00701c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009750 tx=0x7f4ed8009310 comp rx=0 tx=0).stop 2024-03-11T20:48:20.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:20.157+0000 7f4eec875700 1 -- 172.21.15.129:0/19253201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ee8100d10 msgr2=0x7f4ee81976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:48:20.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:20.157+0000 7f4eec875700 1 --2- 172.21.15.129:0/19253201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ee8100d10 0x7f4ee81976c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4edc00f980 tx=0x7f4edc00f9b0 comp rx=0 tx=0).stop 2024-03-11T20:48:20.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:20.157+0000 7f4eec875700 1 -- 172.21.15.129:0/19253201 shutdown_connections 2024-03-11T20:48:20.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:20.157+0000 7f4eec875700 1 --2- 172.21.15.129:0/19253201 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4ed006dd00 0x7f4ed00701c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:20.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:20.157+0000 7f4eec875700 1 --2- 172.21.15.129:0/19253201 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4ee81028b0 0x7f4ee819ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:20.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:20.157+0000 7f4eec875700 1 --2- 172.21.15.129:0/19253201 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ee8100d10 0x7f4ee81976c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:20.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:20.157+0000 7f4eec875700 1 --2- 172.21.15.129:0/19253201 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ee8101f10 0x7f4ee8197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:20.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:20.157+0000 7f4eec875700 1 -- 172.21.15.129:0/19253201 >> 172.21.15.129:0/19253201 conn(0x7f4ee80fc2b0 msgr2=0x7f4ee806a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:48:20.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:20.157+0000 7f4eec875700 1 -- 172.21.15.129:0/19253201 shutdown_connections 2024-03-11T20:48:20.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:20.157+0000 7f4eec875700 1 -- 172.21.15.129:0/19253201 wait complete. 2024-03-11T20:48:20.810 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:20 smithi129 ceph-mon[30184]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:20.810 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:20 smithi129 ceph-mon[30184]: from='osd.1 [v2:172.21.15.129:6810/2519234527,v1:172.21.15.129:6811/2519234527]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T20:48:20.811 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:20 smithi129 ceph-mon[30184]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T20:48:20.811 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:20.811 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:20.811 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:20.811 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:20.811 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:20 smithi129 ceph-mon[42066]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:20.811 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:20 smithi129 ceph-mon[42066]: from='osd.1 [v2:172.21.15.129:6810/2519234527,v1:172.21.15.129:6811/2519234527]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T20:48:20.811 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:20 smithi129 ceph-mon[42066]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T20:48:20.811 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:20 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:20.811 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:20 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:20.811 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:20 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:20.812 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:20 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:20.838 DEBUG:teuthology.orchestra.run.smithi129:osd.1> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T20:48:20.840 INFO:tasks.cephadm:Deploying osd.2 on smithi129 with /dev/vg_nvme/lv_2... 2024-03-11T20:48:20.841 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T20:48:20.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:20 smithi157 ceph-mon[33359]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:20.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:20 smithi157 ceph-mon[33359]: from='osd.1 [v2:172.21.15.129:6810/2519234527,v1:172.21.15.129:6811/2519234527]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T20:48:20.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:20 smithi157 ceph-mon[33359]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T20:48:20.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:20.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:20.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:20.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:21.725 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:48:21.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:21 smithi129 ceph-mon[30184]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T20:48:21.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:21 smithi129 ceph-mon[30184]: from='osd.1 [v2:172.21.15.129:6810/2519234527,v1:172.21.15.129:6811/2519234527]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:48:21.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:21 smithi129 ceph-mon[30184]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T20:48:21.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:21 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:21.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:21 smithi129 ceph-mon[30184]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:48:21.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:21 smithi129 ceph-mon[30184]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-11T20:48:21.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:21 smithi129 ceph-mon[30184]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T20:48:21.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:21 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:21.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:21 smithi129 ceph-mon[42066]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T20:48:21.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:21 smithi129 ceph-mon[42066]: from='osd.1 [v2:172.21.15.129:6810/2519234527,v1:172.21.15.129:6811/2519234527]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:48:21.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:21 smithi129 ceph-mon[42066]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T20:48:21.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:21 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:21.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:21 smithi129 ceph-mon[42066]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:48:21.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:21 smithi129 ceph-mon[42066]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-11T20:48:21.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:21 smithi129 ceph-mon[42066]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T20:48:21.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:21 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:22.041 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:21 smithi157 ceph-mon[33359]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T20:48:22.041 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:21 smithi157 ceph-mon[33359]: from='osd.1 [v2:172.21.15.129:6810/2519234527,v1:172.21.15.129:6811/2519234527]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:48:22.041 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:21 smithi157 ceph-mon[33359]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T20:48:22.041 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:21 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:22.041 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:21 smithi157 ceph-mon[33359]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:48:22.041 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:21 smithi157 ceph-mon[33359]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-11T20:48:22.041 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:21 smithi157 ceph-mon[33359]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T20:48:22.041 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:21 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:22.398 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 20:48:22 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-1[52552]: 2024-03-11T20:48:22.154+0000 7f340c029700 -1 osd.1 0 waiting for initial osdmap 2024-03-11T20:48:22.399 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 20:48:22 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-1[52552]: 2024-03-11T20:48:22.157+0000 7f3403e14700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T20:48:22.931 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:22 smithi129 ceph-mon[30184]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:22.931 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:22 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:22.931 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:22 smithi129 ceph-mon[30184]: from='osd.1 ' entity='osd.1' 2024-03-11T20:48:22.931 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:22 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:22.931 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:22 smithi129 ceph-mon[42066]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:22.931 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:22.931 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:22 smithi129 ceph-mon[42066]: from='osd.1 ' entity='osd.1' 2024-03-11T20:48:22.931 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:23.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:22 smithi157 ceph-mon[33359]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:23.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:22 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:23.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:22 smithi157 ceph-mon[33359]: from='osd.1 ' entity='osd.1' 2024-03-11T20:48:23.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:22 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:23.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:23 smithi129 ceph-mon[30184]: purged_snaps scrub starts 2024-03-11T20:48:23.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:23 smithi129 ceph-mon[30184]: purged_snaps scrub ok 2024-03-11T20:48:23.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:23 smithi129 ceph-mon[30184]: osd.1 [v2:172.21.15.129:6810/2519234527,v1:172.21.15.129:6811/2519234527] boot 2024-03-11T20:48:23.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:23 smithi129 ceph-mon[30184]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T20:48:23.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:23 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:23.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:23 smithi129 ceph-mon[30184]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:23.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:23 smithi129 ceph-mon[42066]: purged_snaps scrub starts 2024-03-11T20:48:23.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:23 smithi129 ceph-mon[42066]: purged_snaps scrub ok 2024-03-11T20:48:23.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:23 smithi129 ceph-mon[42066]: osd.1 [v2:172.21.15.129:6810/2519234527,v1:172.21.15.129:6811/2519234527] boot 2024-03-11T20:48:23.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:23 smithi129 ceph-mon[42066]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T20:48:23.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:23 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:23.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:23 smithi129 ceph-mon[42066]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:24.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:23 smithi157 ceph-mon[33359]: purged_snaps scrub starts 2024-03-11T20:48:24.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:23 smithi157 ceph-mon[33359]: purged_snaps scrub ok 2024-03-11T20:48:24.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:23 smithi157 ceph-mon[33359]: osd.1 [v2:172.21.15.129:6810/2519234527,v1:172.21.15.129:6811/2519234527] boot 2024-03-11T20:48:24.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:23 smithi157 ceph-mon[33359]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T20:48:24.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:23 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:48:24.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:23 smithi157 ceph-mon[33359]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T20:48:25.148 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T20:48:25.148 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T20:48:25.149 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-03-11T20:48:25.149 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-03-11T20:48:25.149 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0146776 s, 714 MB/s 2024-03-11T20:48:25.149 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-03-11T20:48:25.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[42066]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T20:48:25.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[42066]: Detected new or changed devices on smithi129 2024-03-11T20:48:25.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:25.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:25.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:25.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:25.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[42066]: Adjusting osd_memory_target on smithi129 to 8032M 2024-03-11T20:48:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:25.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[30184]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T20:48:25.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[30184]: Detected new or changed devices on smithi129 2024-03-11T20:48:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[30184]: Adjusting osd_memory_target on smithi129 to 8032M 2024-03-11T20:48:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:25.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:25 smithi157 ceph-mon[33359]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T20:48:25.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:25 smithi157 ceph-mon[33359]: Detected new or changed devices on smithi129 2024-03-11T20:48:25.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:25.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:25.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:25.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:25.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:25 smithi157 ceph-mon[33359]: Adjusting osd_memory_target on smithi129 to 8032M 2024-03-11T20:48:25.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:25.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:25.779 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi129:vg_nvme/lv_2 2024-03-11T20:48:26.015 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:48:26.472 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:26 smithi129 ceph-mon[30184]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:26.472 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:26 smithi129 ceph-mon[42066]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:26.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:26 smithi157 ceph-mon[33359]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:27.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.597+0000 7f23e5eb1700 1 -- 172.21.15.129:0/2051660952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23e0100d80 msgr2=0x7f23e01011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:48:27.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.597+0000 7f23e5eb1700 1 --2- 172.21.15.129:0/2051660952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23e0100d80 0x7f23e01011a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f23d40099c0 tx=0x7f23d4009cd0 comp rx=0 tx=0).stop 2024-03-11T20:48:27.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.597+0000 7f23e5eb1700 1 -- 172.21.15.129:0/2051660952 shutdown_connections 2024-03-11T20:48:27.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.597+0000 7f23e5eb1700 1 --2- 172.21.15.129:0/2051660952 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f23e0102920 0x7f23e01091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:27.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.597+0000 7f23e5eb1700 1 --2- 172.21.15.129:0/2051660952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23e0100d80 0x7f23e01011a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:27.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.597+0000 7f23e5eb1700 1 --2- 172.21.15.129:0/2051660952 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23e0101f80 0x7f23e01023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:27.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.597+0000 7f23e5eb1700 1 -- 172.21.15.129:0/2051660952 >> 172.21.15.129:0/2051660952 conn(0x7f23e00fc340 msgr2=0x7f23e00fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:48:27.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.597+0000 7f23e5eb1700 1 -- 172.21.15.129:0/2051660952 shutdown_connections 2024-03-11T20:48:27.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.597+0000 7f23e5eb1700 1 -- 172.21.15.129:0/2051660952 wait complete. 2024-03-11T20:48:27.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23e5eb1700 1 Processor -- start 2024-03-11T20:48:27.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23e5eb1700 1 -- start start 2024-03-11T20:48:27.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23e5eb1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f23e0100d80 0x7f23e006aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:48:27.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23e5eb1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23e0101f80 0x7f23e019c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:48:27.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23e5eb1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23e0102920 0x7f23e019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:48:27.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23e5eb1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e010bd40 con 0x7f23e0102920 2024-03-11T20:48:27.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23e5eb1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e010bbc0 con 0x7f23e0101f80 2024-03-11T20:48:27.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23e5eb1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e010bec0 con 0x7f23e0100d80 2024-03-11T20:48:27.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23df7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f23e0100d80 0x7f23e006aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:48:27.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23df7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f23e0100d80 0x7f23e006aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:47126/0 (socket says 172.21.15.129:47126) 2024-03-11T20:48:27.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23df7fe700 1 -- 172.21.15.129:0/2524987109 learned_addr learned my addr 172.21.15.129:0/2524987109 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:48:27.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23dffff700 1 --2- 172.21.15.129:0/2524987109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23e0102920 0x7f23e019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:48:27.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23deffd700 1 --2- 172.21.15.129:0/2524987109 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23e0101f80 0x7f23e019c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:48:27.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23df7fe700 1 -- 172.21.15.129:0/2524987109 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23e0101f80 msgr2=0x7f23e019c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:48:27.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23df7fe700 1 --2- 172.21.15.129:0/2524987109 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23e0101f80 0x7f23e019c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:27.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23df7fe700 1 -- 172.21.15.129:0/2524987109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23e0102920 msgr2=0x7f23e019e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:48:27.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23df7fe700 1 --2- 172.21.15.129:0/2524987109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23e0102920 0x7f23e019e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:27.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23df7fe700 1 -- 172.21.15.129:0/2524987109 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e019f1f0 con 0x7f23e0100d80 2024-03-11T20:48:27.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.598+0000 7f23dffff700 1 --2- 172.21.15.129:0/2524987109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23e0102920 0x7f23e019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:48:27.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.599+0000 7f23df7fe700 1 --2- 172.21.15.129:0/2524987109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f23e0100d80 0x7f23e006aff0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f23d400b580 tx=0x7f23d40056a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:48:27.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.599+0000 7f23dcff9700 1 -- 172.21.15.129:0/2524987109 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d4026070 con 0x7f23e0100d80 2024-03-11T20:48:27.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.599+0000 7f23dcff9700 1 -- 172.21.15.129:0/2524987109 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f23d4003e50 con 0x7f23e0100d80 2024-03-11T20:48:27.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.599+0000 7f23e5eb1700 1 -- 172.21.15.129:0/2524987109 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e019f480 con 0x7f23e0100d80 2024-03-11T20:48:27.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.599+0000 7f23dcff9700 1 -- 172.21.15.129:0/2524987109 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d400fb30 con 0x7f23e0100d80 2024-03-11T20:48:27.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.599+0000 7f23e5eb1700 1 -- 172.21.15.129:0/2524987109 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e019f8d0 con 0x7f23e0100d80 2024-03-11T20:48:27.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.600+0000 7f23e5eb1700 1 -- 172.21.15.129:0/2524987109 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e0068aa0 con 0x7f23e0100d80 2024-03-11T20:48:27.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.600+0000 7f23dcff9700 1 -- 172.21.15.129:0/2524987109 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 14) v1 ==== 91699+0+0 (secure 0 0 0) 0x7f23d4029070 con 0x7f23e0100d80 2024-03-11T20:48:27.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.601+0000 7f23dcff9700 1 --2- 172.21.15.129:0/2524987109 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f23c806dd00 0x7f23c80701c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:48:27.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.601+0000 7f23dcff9700 1 -- 172.21.15.129:0/2524987109 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2199+0+0 (secure 0 0 0) 0x7f23d408d5f0 con 0x7f23e0100d80 2024-03-11T20:48:27.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.601+0000 7f23deffd700 1 --2- 172.21.15.129:0/2524987109 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f23c806dd00 0x7f23c80701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:48:27.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.601+0000 7f23deffd700 1 --2- 172.21.15.129:0/2524987109 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f23c806dd00 0x7f23c80701c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f23d0005ea0 tx=0x7f23d0005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:48:27.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.606+0000 7f23dcff9700 1 -- 172.21.15.129:0/2524987109 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f23d4013070 con 0x7f23e0100d80 2024-03-11T20:48:27.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:27.814+0000 7f23e5eb1700 1 -- 172.21.15.129:0/2524987109 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f23e002dc70 con 0x7f23c806dd00 2024-03-11T20:48:28.672 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:28 smithi129 ceph-mon[30184]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:28.672 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:28 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:48:28.672 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:28 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:48:28.672 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:28 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:28.672 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:28 smithi129 ceph-mon[42066]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:28.672 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:28 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:48:28.672 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:28 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:48:28.672 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:28 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:28.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:28 smithi157 ceph-mon[33359]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:28.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:28 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:48:28.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:28 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:48:28.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:28 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:29.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:29 smithi129 ceph-mon[30184]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:48:29.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:29 smithi129 ceph-mon[42066]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:48:29.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:29 smithi157 ceph-mon[33359]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:48:30.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:30 smithi129 ceph-mon[30184]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:30.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3509743866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b6db120-1132-4369-9f17-777545ad4a07"}]: dispatch 2024-03-11T20:48:30.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3509743866' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b6db120-1132-4369-9f17-777545ad4a07"}]': finished 2024-03-11T20:48:30.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:30 smithi129 ceph-mon[30184]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T20:48:30.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:30 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:30.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:30 smithi129 ceph-mon[42066]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:30.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3509743866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b6db120-1132-4369-9f17-777545ad4a07"}]: dispatch 2024-03-11T20:48:30.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3509743866' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b6db120-1132-4369-9f17-777545ad4a07"}]': finished 2024-03-11T20:48:30.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:30 smithi129 ceph-mon[42066]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T20:48:30.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:30 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:30.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:30 smithi157 ceph-mon[33359]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:30.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3509743866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b6db120-1132-4369-9f17-777545ad4a07"}]: dispatch 2024-03-11T20:48:30.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3509743866' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b6db120-1132-4369-9f17-777545ad4a07"}]': finished 2024-03-11T20:48:30.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:30 smithi157 ceph-mon[33359]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T20:48:30.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:30 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:31.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2929826265' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:48:31.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2929826265' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:48:31.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2929826265' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:48:32.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:32 smithi129 ceph-mon[30184]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:32.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:32 smithi129 ceph-mon[42066]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:32.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:32 smithi157 ceph-mon[33359]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:34.894 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:34 smithi129 ceph-mon[30184]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:34.895 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:34 smithi129 ceph-mon[42066]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:34.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:34 smithi157 ceph-mon[33359]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:36.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:36 smithi129 ceph-mon[30184]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:36.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:36 smithi129 ceph-mon[42066]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:36.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:36 smithi157 ceph-mon[33359]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:37.873 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:37 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T20:48:37.873 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:37 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:37.874 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:37 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T20:48:37.874 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:37 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:37.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:37 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T20:48:37.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:37 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:38.927 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:38 smithi129 ceph-mon[30184]: Deploying daemon osd.2 on smithi129 2024-03-11T20:48:38.928 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:38 smithi129 ceph-mon[30184]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:38.928 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:38 smithi129 ceph-mon[42066]: Deploying daemon osd.2 on smithi129 2024-03-11T20:48:38.928 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:38 smithi129 ceph-mon[42066]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:38.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:38 smithi157 ceph-mon[33359]: Deploying daemon osd.2 on smithi129 2024-03-11T20:48:38.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:38 smithi157 ceph-mon[33359]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:40.927 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:40 smithi129 ceph-mon[30184]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:40.927 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:40 smithi129 ceph-mon[42066]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:40.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:40 smithi157 ceph-mon[33359]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:42.947 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:42 smithi129 ceph-mon[30184]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:42.947 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:42 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:42.947 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:42 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:42.948 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:42 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:42.948 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:42 smithi129 ceph-mon[42066]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:42.948 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:42 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:42.948 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:42 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:42.948 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:42 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:42.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:42 smithi157 ceph-mon[33359]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:42.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:42 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:42.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:42 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:42.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:42 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:44.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:44 smithi129 ceph-mon[30184]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:44.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:44 smithi129 ceph-mon[42066]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:44.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:44 smithi157 ceph-mon[33359]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:46.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:46.466+0000 7f23dcff9700 1 -- 172.21.15.129:0/2524987109 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f23e002dc70 con 0x7f23c806dd00 2024-03-11T20:48:46.465 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 2 on host 'smithi129' 2024-03-11T20:48:46.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:46.468+0000 7f23e5eb1700 1 -- 172.21.15.129:0/2524987109 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f23c806dd00 msgr2=0x7f23c80701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:48:46.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:46.468+0000 7f23e5eb1700 1 --2- 172.21.15.129:0/2524987109 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f23c806dd00 0x7f23c80701c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f23d0005ea0 tx=0x7f23d0005e30 comp rx=0 tx=0).stop 2024-03-11T20:48:46.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:46.468+0000 7f23e5eb1700 1 -- 172.21.15.129:0/2524987109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f23e0100d80 msgr2=0x7f23e006aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:48:46.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:46.468+0000 7f23e5eb1700 1 --2- 172.21.15.129:0/2524987109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f23e0100d80 0x7f23e006aff0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f23d400b580 tx=0x7f23d40056a0 comp rx=0 tx=0).stop 2024-03-11T20:48:46.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:46.468+0000 7f23e5eb1700 1 -- 172.21.15.129:0/2524987109 shutdown_connections 2024-03-11T20:48:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:46.468+0000 7f23e5eb1700 1 --2- 172.21.15.129:0/2524987109 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f23c806dd00 0x7f23c80701c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:46.468+0000 7f23e5eb1700 1 --2- 172.21.15.129:0/2524987109 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23e0102920 0x7f23e019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:46.468+0000 7f23e5eb1700 1 --2- 172.21.15.129:0/2524987109 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f23e0100d80 0x7f23e006aff0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:46.468+0000 7f23e5eb1700 1 --2- 172.21.15.129:0/2524987109 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23e0101f80 0x7f23e019c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:46.469+0000 7f23e5eb1700 1 -- 172.21.15.129:0/2524987109 >> 172.21.15.129:0/2524987109 conn(0x7f23e00fc340 msgr2=0x7f23e00fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:48:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:46.469+0000 7f23e5eb1700 1 -- 172.21.15.129:0/2524987109 shutdown_connections 2024-03-11T20:48:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:46.469+0000 7f23e5eb1700 1 -- 172.21.15.129:0/2524987109 wait complete. 2024-03-11T20:48:46.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:46.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:46.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[42066]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:46.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[42066]: from='osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T20:48:46.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:46.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:46.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:46.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:46.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:46.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:46.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:46.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:46.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[30184]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:46.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[30184]: from='osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T20:48:46.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:46.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:46.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:46.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:46.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:46.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:46.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:46 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:46.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:46 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:46.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:46 smithi157 ceph-mon[33359]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:46.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:46 smithi157 ceph-mon[33359]: from='osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T20:48:46.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:46 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:46.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:46 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:46.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:46 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:46.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:46 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:46.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:46 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:46.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:46 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:48:47.045 DEBUG:teuthology.orchestra.run.smithi129:osd.2> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T20:48:47.047 INFO:tasks.cephadm:Deploying osd.3 on smithi129 with /dev/vg_nvme/lv_1... 2024-03-11T20:48:47.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T20:48:47.831 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:48:47.955 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:47 smithi129 ceph-mon[30184]: from='osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T20:48:47.955 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:47 smithi129 ceph-mon[30184]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T20:48:47.955 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:47 smithi129 ceph-mon[30184]: from='osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:48:47.955 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:47 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:47.955 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:47 smithi129 ceph-mon[42066]: from='osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T20:48:47.955 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:47 smithi129 ceph-mon[42066]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T20:48:47.955 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:47 smithi129 ceph-mon[42066]: from='osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:48:47.955 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:47 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:47.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:47 smithi157 ceph-mon[33359]: from='osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T20:48:47.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:47 smithi157 ceph-mon[33359]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T20:48:47.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:47 smithi157 ceph-mon[33359]: from='osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:48:47.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:47 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:48.209 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 20:48:48 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-2[58560]: 2024-03-11T20:48:48.011+0000 7f3b669cc700 -1 osd.2 0 waiting for initial osdmap 2024-03-11T20:48:48.209 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 20:48:48 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-2[58560]: 2024-03-11T20:48:48.018+0000 7f3b5e7b7700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T20:48:48.744 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:48 smithi129 ceph-mon[30184]: from='osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-11T20:48:48.744 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:48 smithi129 ceph-mon[30184]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T20:48:48.744 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:48 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:48.744 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:48 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:48.744 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:48 smithi129 ceph-mon[30184]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:48.744 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:48 smithi129 ceph-mon[30184]: from='osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087]' entity='osd.2' 2024-03-11T20:48:48.744 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:48 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:48.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:48 smithi129 ceph-mon[42066]: from='osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-11T20:48:48.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:48 smithi129 ceph-mon[42066]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T20:48:48.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:48 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:48.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:48 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:48.744 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:48 smithi129 ceph-mon[42066]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:48.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:48 smithi129 ceph-mon[42066]: from='osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087]' entity='osd.2' 2024-03-11T20:48:48.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:48 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:48.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:48 smithi157 ceph-mon[33359]: from='osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-11T20:48:48.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:48 smithi157 ceph-mon[33359]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T20:48:48.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:48 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:48.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:48 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:48.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:48 smithi157 ceph-mon[33359]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T20:48:48.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:48 smithi157 ceph-mon[33359]: from='osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087]' entity='osd.2' 2024-03-11T20:48:48.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:48 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:49.784 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:49 smithi129 ceph-mon[30184]: purged_snaps scrub starts 2024-03-11T20:48:49.784 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:49 smithi129 ceph-mon[30184]: purged_snaps scrub ok 2024-03-11T20:48:49.784 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:49 smithi129 ceph-mon[30184]: osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087] boot 2024-03-11T20:48:49.785 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:49 smithi129 ceph-mon[30184]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T20:48:49.785 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:49 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:49.785 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:49 smithi129 ceph-mon[42066]: purged_snaps scrub starts 2024-03-11T20:48:49.785 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:49 smithi129 ceph-mon[42066]: purged_snaps scrub ok 2024-03-11T20:48:49.785 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:49 smithi129 ceph-mon[42066]: osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087] boot 2024-03-11T20:48:49.785 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:49 smithi129 ceph-mon[42066]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T20:48:49.785 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:49 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:49.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:49 smithi157 ceph-mon[33359]: purged_snaps scrub starts 2024-03-11T20:48:49.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:49 smithi157 ceph-mon[33359]: purged_snaps scrub ok 2024-03-11T20:48:49.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:49 smithi157 ceph-mon[33359]: osd.2 [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087] boot 2024-03-11T20:48:49.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:49 smithi157 ceph-mon[33359]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T20:48:49.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:49 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:48:50.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[30184]: pgmap v78: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:48:50.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:48:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T20:48:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[30184]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T20:48:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:48:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[42066]: pgmap v78: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:48:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:48:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T20:48:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[42066]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T20:48:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:48:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:50 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:50.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:50 smithi157 ceph-mon[33359]: pgmap v78: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:48:50.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:50 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:48:50.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:50 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T20:48:50.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:50 smithi157 ceph-mon[33359]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T20:48:50.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:50 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:48:50.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:50 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:50.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:50 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:50.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:50 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:50.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:50 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:50.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:50 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:48:50.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:50 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:50.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:50 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:50.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:50 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:48:50.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:50 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:51.220 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T20:48:51.221 INFO:teuthology.orchestra.run.smithi129.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T20:48:51.221 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10+0 records in 2024-03-11T20:48:51.221 INFO:teuthology.orchestra.run.smithi129.stderr:10+0 records out 2024-03-11T20:48:51.221 INFO:teuthology.orchestra.run.smithi129.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129773 s, 808 MB/s 2024-03-11T20:48:51.221 INFO:teuthology.orchestra.run.smithi129.stderr:--> Zapping successful for: 2024-03-11T20:48:51.855 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:51 smithi129 ceph-mon[30184]: Detected new or changed devices on smithi129 2024-03-11T20:48:51.855 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:51 smithi129 ceph-mon[30184]: Adjusting osd_memory_target on smithi129 to 5354M 2024-03-11T20:48:51.855 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:51 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:48:51.855 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:51 smithi129 ceph-mon[30184]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T20:48:51.855 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:51 smithi129 ceph-mon[42066]: Detected new or changed devices on smithi129 2024-03-11T20:48:51.856 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:51 smithi129 ceph-mon[42066]: Adjusting osd_memory_target on smithi129 to 5354M 2024-03-11T20:48:51.856 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:51 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:48:51.856 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:51 smithi129 ceph-mon[42066]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T20:48:51.949 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi129:vg_nvme/lv_1 2024-03-11T20:48:51.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:51 smithi157 ceph-mon[33359]: Detected new or changed devices on smithi129 2024-03-11T20:48:51.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:51 smithi157 ceph-mon[33359]: Adjusting osd_memory_target on smithi129 to 5354M 2024-03-11T20:48:51.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:51 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:48:51.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:51 smithi157 ceph-mon[33359]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T20:48:52.120 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 20:48:51 smithi129 sudo[62778]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T20:48:52.120 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 20:48:51 smithi129 sudo[62778]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T20:48:52.120 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 20:48:51 smithi129 sudo[62778]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T20:48:52.120 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 20:48:51 smithi129 sudo[62778]: pam_unix(sudo:session): session closed for user root 2024-03-11T20:48:52.203 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:48:52.405 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 20:48:52 smithi129 sudo[62803]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T20:48:52.405 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 20:48:52 smithi129 sudo[62803]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T20:48:52.406 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 20:48:52 smithi129 sudo[62803]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T20:48:52.406 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 20:48:52 smithi129 sudo[62803]: pam_unix(sudo:session): session closed for user root 2024-03-11T20:48:52.675 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:52 smithi129 ceph-mon[30184]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:48:52.675 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:52 smithi129 ceph-mon[30184]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T20:48:52.675 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:52 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:52.675 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:52 smithi129 ceph-mon[42066]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:48:52.675 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:52 smithi129 ceph-mon[42066]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T20:48:52.675 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:52 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:52.675 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 20:48:52 smithi129 sudo[62856]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T20:48:52.676 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 20:48:52 smithi129 sudo[62856]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T20:48:52.676 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 20:48:52 smithi129 sudo[62856]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T20:48:52.676 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 20:48:52 smithi129 sudo[62856]: pam_unix(sudo:session): session closed for user root 2024-03-11T20:48:52.935 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 20:48:52 smithi129 sudo[62899]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T20:48:52.935 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 20:48:52 smithi129 sudo[62899]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T20:48:52.935 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 20:48:52 smithi129 sudo[62899]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T20:48:52.935 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 20:48:52 smithi129 sudo[62899]: pam_unix(sudo:session): session closed for user root 2024-03-11T20:48:52.935 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 20:48:52 smithi129 sudo[62902]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T20:48:52.936 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 20:48:52 smithi129 sudo[62902]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T20:48:52.936 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 20:48:52 smithi129 sudo[62902]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T20:48:52.936 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 20:48:52 smithi129 sudo[62902]: pam_unix(sudo:session): session closed for user root 2024-03-11T20:48:52.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:52 smithi157 ceph-mon[33359]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:48:52.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:52 smithi157 ceph-mon[33359]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T20:48:52.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:52 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:53.227 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 20:48:53 smithi129 sudo[62912]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T20:48:53.228 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 20:48:53 smithi129 sudo[62912]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T20:48:53.228 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 20:48:53 smithi129 sudo[62912]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T20:48:53.228 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 20:48:53 smithi129 sudo[62912]: pam_unix(sudo:session): session closed for user root 2024-03-11T20:48:53.649 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:53 smithi129 sudo[62935]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T20:48:53.649 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:53 smithi129 sudo[62935]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T20:48:53.649 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:53 smithi129 sudo[62935]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T20:48:53.649 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:53 smithi129 sudo[62935]: pam_unix(sudo:session): session closed for user root 2024-03-11T20:48:53.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.892+0000 7fb74cae1700 1 -- 172.21.15.129:0/1198003835 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7481028a0 msgr2=0x7fb748109150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:48:53.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.892+0000 7fb74cae1700 1 --2- 172.21.15.129:0/1198003835 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7481028a0 0x7fb748109150 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb74000b0a0 tx=0x7fb74000b3b0 comp rx=0 tx=0).stop 2024-03-11T20:48:53.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.893+0000 7fb74cae1700 1 -- 172.21.15.129:0/1198003835 shutdown_connections 2024-03-11T20:48:53.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.893+0000 7fb74cae1700 1 --2- 172.21.15.129:0/1198003835 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7481028a0 0x7fb748109150 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:53.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.893+0000 7fb74cae1700 1 --2- 172.21.15.129:0/1198003835 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb748100d00 0x7fb748101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:53.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.893+0000 7fb74cae1700 1 --2- 172.21.15.129:0/1198003835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb748101f00 0x7fb748102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:53.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.893+0000 7fb74cae1700 1 -- 172.21.15.129:0/1198003835 >> 172.21.15.129:0/1198003835 conn(0x7fb7480fc2e0 msgr2=0x7fb7480fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:48:53.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.893+0000 7fb74cae1700 1 -- 172.21.15.129:0/1198003835 shutdown_connections 2024-03-11T20:48:53.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.893+0000 7fb74cae1700 1 -- 172.21.15.129:0/1198003835 wait complete. 2024-03-11T20:48:53.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.893+0000 7fb74cae1700 1 Processor -- start 2024-03-11T20:48:53.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.893+0000 7fb74cae1700 1 -- start start 2024-03-11T20:48:53.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.893+0000 7fb74cae1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb748100d00 0x7fb74806b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:48:53.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.893+0000 7fb74cae1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb748101f00 0x7fb74819c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:48:53.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.893+0000 7fb74cae1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7481028a0 0x7fb74819e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:48:53.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb74cae1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74810bd90 con 0x7fb748100d00 2024-03-11T20:48:53.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb74cae1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74810bc10 con 0x7fb748101f00 2024-03-11T20:48:53.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb74cae1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74810bf10 con 0x7fb7481028a0 2024-03-11T20:48:53.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb746d9d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7481028a0 0x7fb74819e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:48:53.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb746d9d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7481028a0 0x7fb74819e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:41472/0 (socket says 172.21.15.129:41472) 2024-03-11T20:48:53.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb746d9d700 1 -- 172.21.15.129:0/4204845606 learned_addr learned my addr 172.21.15.129:0/4204845606 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:48:53.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb745d9b700 1 --2- 172.21.15.129:0/4204845606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb748101f00 0x7fb74819c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:48:53.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb74659c700 1 --2- 172.21.15.129:0/4204845606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb748100d00 0x7fb74806b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:48:53.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb746d9d700 1 -- 172.21.15.129:0/4204845606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb748101f00 msgr2=0x7fb74819c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:48:53.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb746d9d700 1 --2- 172.21.15.129:0/4204845606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb748101f00 0x7fb74819c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:53.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb746d9d700 1 -- 172.21.15.129:0/4204845606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb748100d00 msgr2=0x7fb74806b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:48:53.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb746d9d700 1 --2- 172.21.15.129:0/4204845606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb748100d00 0x7fb74806b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:48:53.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb746d9d700 1 -- 172.21.15.129:0/4204845606 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74819f130 con 0x7fb7481028a0 2024-03-11T20:48:53.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb746d9d700 1 --2- 172.21.15.129:0/4204845606 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7481028a0 0x7fb74819e860 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb74000b780 tx=0x7fb740004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:48:53.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb7377fe700 1 -- 172.21.15.129:0/4204845606 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb740021030 con 0x7fb7481028a0 2024-03-11T20:48:53.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb7377fe700 1 -- 172.21.15.129:0/4204845606 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb740003a00 con 0x7fb7481028a0 2024-03-11T20:48:53.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb74cae1700 1 -- 172.21.15.129:0/4204845606 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74819f3c0 con 0x7fb7481028a0 2024-03-11T20:48:53.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb7377fe700 1 -- 172.21.15.129:0/4204845606 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb740004440 con 0x7fb7481028a0 2024-03-11T20:48:53.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.894+0000 7fb74cae1700 1 -- 172.21.15.129:0/4204845606 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74819f870 con 0x7fb7481028a0 2024-03-11T20:48:53.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.895+0000 7fb74cae1700 1 -- 172.21.15.129:0/4204845606 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb748068aa0 con 0x7fb7481028a0 2024-03-11T20:48:53.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.896+0000 7fb7377fe700 1 -- 172.21.15.129:0/4204845606 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7fb740024070 con 0x7fb7481028a0 2024-03-11T20:48:53.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.896+0000 7fb7377fe700 1 --2- 172.21.15.129:0/4204845606 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb73006dd90 0x7fb730070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:48:53.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.896+0000 7fb7377fe700 1 -- 172.21.15.129:0/4204845606 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3017+0+0 (secure 0 0 0) 0x7fb74000f020 con 0x7fb7481028a0 2024-03-11T20:48:53.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.896+0000 7fb74659c700 1 --2- 172.21.15.129:0/4204845606 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb73006dd90 0x7fb730070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:48:53.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.897+0000 7fb74659c700 1 --2- 172.21.15.129:0/4204845606 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb73006dd90 0x7fb730070250 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb748101d60 tx=0x7fb73800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:48:53.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:53.901+0000 7fb7377fe700 1 -- 172.21.15.129:0/4204845606 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fb74005bf50 con 0x7fb7481028a0 2024-03-11T20:48:53.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:53 smithi129 sudo[62958]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T20:48:53.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:53 smithi129 sudo[62958]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T20:48:53.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:53 smithi129 sudo[62958]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T20:48:53.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:53 smithi129 sudo[62958]: pam_unix(sudo:session): session closed for user root 2024-03-11T20:48:54.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:48:54.104+0000 7fb74cae1700 1 -- 172.21.15.129:0/4204845606 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb748062220 con 0x7fb73006dd90 2024-03-11T20:48:54.194 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:53 smithi157 sudo[36203]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T20:48:54.194 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:53 smithi157 sudo[36203]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T20:48:54.194 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:53 smithi157 sudo[36203]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T20:48:54.194 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:53 smithi157 sudo[36203]: pam_unix(sudo:session): session closed for user root 2024-03-11T20:48:54.472 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[30184]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-11T20:48:54.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[30184]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:48:54.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[30184]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:48:54.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[30184]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:48:54.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:48:54.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:48:54.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:48:54.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:48:54.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:48:54.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:48:54.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:48:54.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:48:54.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:54.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[42066]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-11T20:48:54.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[42066]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:48:54.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[42066]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:48:54.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[42066]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:48:54.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:48:54.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:48:54.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:48:54.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:48:54.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:48:54.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:48:54.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:48:54.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:48:54.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:54.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 sudo[36206]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T20:48:54.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 sudo[36206]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T20:48:54.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 sudo[36206]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T20:48:54.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 sudo[36206]: pam_unix(sudo:session): session closed for user root 2024-03-11T20:48:54.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 ceph-mon[33359]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-11T20:48:54.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 ceph-mon[33359]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:48:54.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 ceph-mon[33359]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:48:54.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 ceph-mon[33359]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:48:54.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:48:54.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:48:54.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:48:54.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:48:54.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:48:54.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:48:54.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:48:54.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:48:54.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:48:54.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 sudo[63055]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T20:48:54.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 sudo[63055]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T20:48:54.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 sudo[63055]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T20:48:54.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 sudo[63055]: pam_unix(sudo:session): session closed for user root 2024-03-11T20:48:54.985 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 sudo[63113]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T20:48:54.985 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 sudo[63113]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T20:48:54.985 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 sudo[63113]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T20:48:54.985 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:54 smithi129 sudo[63113]: pam_unix(sudo:session): session closed for user root 2024-03-11T20:48:55.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[42066]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:48:55.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[42066]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:48:55.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[42066]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:48:55.467 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:55.467 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[42066]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:48:55.467 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:48:55.467 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:48:55.467 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:48:55.467 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[42066]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:48:55.467 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[30184]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:48:55.467 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[30184]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:48:55.467 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[30184]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:48:55.467 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:55.467 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[30184]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:48:55.468 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:48:55.468 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:48:55.468 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:48:55.468 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:55 smithi129 ceph-mon[30184]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:48:55.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:55 smithi157 ceph-mon[33359]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:48:55.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:55 smithi157 ceph-mon[33359]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:48:55.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:55 smithi157 ceph-mon[33359]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:48:55.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:55 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:48:55.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:55 smithi157 ceph-mon[33359]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T20:48:55.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:55 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:48:55.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:55 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:48:55.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:55 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:48:55.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:55 smithi157 ceph-mon[33359]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T20:48:56.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:56 smithi129 ceph-mon[30184]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:48:56.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:56 smithi129 ceph-mon[42066]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:48:56.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:56 smithi157 ceph-mon[33359]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:48:57.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/161261079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9d3db4d-385d-4745-b71f-0796ff551f4b"}]: dispatch 2024-03-11T20:48:57.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9d3db4d-385d-4745-b71f-0796ff551f4b"}]: dispatch 2024-03-11T20:48:57.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9d3db4d-385d-4745-b71f-0796ff551f4b"}]': finished 2024-03-11T20:48:57.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:57 smithi129 ceph-mon[30184]: osdmap e22: 4 total, 3 up, 4 in 2024-03-11T20:48:57.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:57 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:48:57.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/161261079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9d3db4d-385d-4745-b71f-0796ff551f4b"}]: dispatch 2024-03-11T20:48:57.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9d3db4d-385d-4745-b71f-0796ff551f4b"}]: dispatch 2024-03-11T20:48:57.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9d3db4d-385d-4745-b71f-0796ff551f4b"}]': finished 2024-03-11T20:48:57.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:57 smithi129 ceph-mon[42066]: osdmap e22: 4 total, 3 up, 4 in 2024-03-11T20:48:57.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:57 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:48:57.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/161261079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9d3db4d-385d-4745-b71f-0796ff551f4b"}]: dispatch 2024-03-11T20:48:57.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9d3db4d-385d-4745-b71f-0796ff551f4b"}]: dispatch 2024-03-11T20:48:57.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9d3db4d-385d-4745-b71f-0796ff551f4b"}]': finished 2024-03-11T20:48:57.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:57 smithi157 ceph-mon[33359]: osdmap e22: 4 total, 3 up, 4 in 2024-03-11T20:48:57.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:57 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:48:58.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2046953933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:48:58.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:48:58 smithi129 ceph-mon[30184]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:48:58.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2046953933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:48:58.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:48:58 smithi129 ceph-mon[42066]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:48:58.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2046953933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:48:58.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:48:58 smithi157 ceph-mon[33359]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:49:00.972 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:00 smithi129 ceph-mon[30184]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-11T20:49:00.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:00 smithi129 ceph-mon[42066]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-11T20:49:00.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:00 smithi157 ceph-mon[33359]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-11T20:49:02.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:02 smithi129 ceph-mon[30184]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T20:49:02.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:02 smithi129 ceph-mon[42066]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T20:49:02.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:02 smithi157 ceph-mon[33359]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T20:49:03.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:03 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T20:49:03.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:03 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:03.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:03 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T20:49:03.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:03 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:03.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:03 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T20:49:03.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:03 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:04.973 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:04 smithi129 ceph-mon[30184]: Deploying daemon osd.3 on smithi129 2024-03-11T20:49:04.973 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:04 smithi129 ceph-mon[30184]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-11T20:49:04.974 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:04 smithi129 ceph-mon[42066]: Deploying daemon osd.3 on smithi129 2024-03-11T20:49:04.974 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:04 smithi129 ceph-mon[42066]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-11T20:49:04.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:04 smithi157 ceph-mon[33359]: Deploying daemon osd.3 on smithi129 2024-03-11T20:49:04.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:04 smithi157 ceph-mon[33359]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-11T20:49:06.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:06 smithi129 ceph-mon[30184]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-11T20:49:06.915 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:06 smithi129 ceph-mon[42066]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-11T20:49:06.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:06 smithi157 ceph-mon[33359]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-11T20:49:08.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:08 smithi129 ceph-mon[30184]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 73 KiB/s wr, 5 op/s 2024-03-11T20:49:08.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:08 smithi129 ceph-mon[42066]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 73 KiB/s wr, 5 op/s 2024-03-11T20:49:08.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:08 smithi157 ceph-mon[33359]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 73 KiB/s wr, 5 op/s 2024-03-11T20:49:10.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:10 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:10.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:10 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:10.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:10 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:10.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:10 smithi129 ceph-mon[30184]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T20:49:10.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:10 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:10.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:10 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:10 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:10 smithi129 ceph-mon[42066]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T20:49:10.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:10 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:10.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:10 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:10.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:10 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:10.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:10 smithi157 ceph-mon[33359]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T20:49:12.857 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:12 smithi129 ceph-mon[30184]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:49:12.857 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:12 smithi129 ceph-mon[30184]: from='osd.3 [v2:172.21.15.129:6826/2265856698,v1:172.21.15.129:6827/2265856698]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T20:49:12.857 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:12 smithi129 ceph-mon[30184]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T20:49:12.857 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:12 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:12.857 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:12 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:12.857 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:12 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:12.857 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:12 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:12.857 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:12 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:12.858 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:12 smithi129 ceph-mon[42066]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:49:12.858 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:12 smithi129 ceph-mon[42066]: from='osd.3 [v2:172.21.15.129:6826/2265856698,v1:172.21.15.129:6827/2265856698]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T20:49:12.858 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:12 smithi129 ceph-mon[42066]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T20:49:12.858 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:12 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:12.858 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:12 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:12.858 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:12 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:12.858 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:12 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:12.858 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:12 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:12.907 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 3 on host 'smithi129' 2024-03-11T20:49:12.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:49:12.911+0000 7fb7377fe700 1 -- 172.21.15.129:0/4204845606 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb748062220 con 0x7fb73006dd90 2024-03-11T20:49:12.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:49:12.914+0000 7fb74cae1700 1 -- 172.21.15.129:0/4204845606 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb73006dd90 msgr2=0x7fb730070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:49:12.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:49:12.914+0000 7fb74cae1700 1 --2- 172.21.15.129:0/4204845606 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb73006dd90 0x7fb730070250 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb748101d60 tx=0x7fb73800b410 comp rx=0 tx=0).stop 2024-03-11T20:49:12.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:49:12.914+0000 7fb74cae1700 1 -- 172.21.15.129:0/4204845606 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7481028a0 msgr2=0x7fb74819e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:49:12.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:49:12.914+0000 7fb74cae1700 1 --2- 172.21.15.129:0/4204845606 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7481028a0 0x7fb74819e860 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb74000b780 tx=0x7fb740004d00 comp rx=0 tx=0).stop 2024-03-11T20:49:12.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:49:12.914+0000 7fb74cae1700 1 -- 172.21.15.129:0/4204845606 shutdown_connections 2024-03-11T20:49:12.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:49:12.914+0000 7fb74cae1700 1 --2- 172.21.15.129:0/4204845606 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb73006dd90 0x7fb730070250 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:12.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:49:12.914+0000 7fb74cae1700 1 --2- 172.21.15.129:0/4204845606 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7481028a0 0x7fb74819e860 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:12.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:49:12.914+0000 7fb74cae1700 1 --2- 172.21.15.129:0/4204845606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb748100d00 0x7fb74806b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:12.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:49:12.914+0000 7fb74cae1700 1 --2- 172.21.15.129:0/4204845606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb748101f00 0x7fb74819c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:12.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:49:12.914+0000 7fb74cae1700 1 -- 172.21.15.129:0/4204845606 >> 172.21.15.129:0/4204845606 conn(0x7fb7480fc2e0 msgr2=0x7fb7480fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:49:12.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:49:12.914+0000 7fb74cae1700 1 -- 172.21.15.129:0/4204845606 shutdown_connections 2024-03-11T20:49:12.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:49:12.914+0000 7fb74cae1700 1 -- 172.21.15.129:0/4204845606 wait complete. 2024-03-11T20:49:12.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:12 smithi157 ceph-mon[33359]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:49:12.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:12 smithi157 ceph-mon[33359]: from='osd.3 [v2:172.21.15.129:6826/2265856698,v1:172.21.15.129:6827/2265856698]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T20:49:12.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:12 smithi157 ceph-mon[33359]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T20:49:12.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:12 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:12.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:12 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:12.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:12 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:12.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:12 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:12.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:12 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:13.570 DEBUG:teuthology.orchestra.run.smithi129:osd.3> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.3.service 2024-03-11T20:49:13.572 INFO:tasks.cephadm:Deploying osd.4 on smithi157 with /dev/vg_nvme/lv_4... 2024-03-11T20:49:13.572 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T20:49:13.819 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:49:13.869 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-mon[30184]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T20:49:13.869 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-mon[30184]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T20:49:13.869 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:49:13.869 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-mon[30184]: from='osd.3 [v2:172.21.15.129:6826/2265856698,v1:172.21.15.129:6827/2265856698]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:49:13.869 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-mon[30184]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:49:13.869 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:13.869 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:13.869 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:13.870 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3[65041]: 2024-03-11T20:49:13.667+0000 7f53a8e40700 -1 osd.3 0 waiting for initial osdmap 2024-03-11T20:49:13.870 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3[65041]: 2024-03-11T20:49:13.670+0000 7f53a4c33700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T20:49:13.870 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-mon[42066]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T20:49:13.870 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-mon[42066]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T20:49:13.870 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:49:13.870 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-mon[42066]: from='osd.3 [v2:172.21.15.129:6826/2265856698,v1:172.21.15.129:6827/2265856698]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:49:13.870 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-mon[42066]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:49:13.870 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:13.870 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:13.871 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:13 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:13.942 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:13 smithi157 ceph-mon[33359]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T20:49:13.942 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:13 smithi157 ceph-mon[33359]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T20:49:13.942 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:13 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:49:13.942 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:13 smithi157 ceph-mon[33359]: from='osd.3 [v2:172.21.15.129:6826/2265856698,v1:172.21.15.129:6827/2265856698]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:49:13.942 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:13 smithi157 ceph-mon[33359]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-03-11T20:49:13.943 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:13 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:13.943 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:13 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:13.943 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:13 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:14.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:14 smithi129 ceph-mon[30184]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:49:14.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:14 smithi129 ceph-mon[30184]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-11T20:49:14.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:14 smithi129 ceph-mon[30184]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T20:49:14.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:14 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:49:14.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:14 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:49:14.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:14 smithi129 ceph-mon[42066]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:49:14.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:14 smithi129 ceph-mon[42066]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-11T20:49:14.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:14 smithi129 ceph-mon[42066]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T20:49:14.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:14 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:49:14.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:14 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:49:14.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:14 smithi157 ceph-mon[33359]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:49:14.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:14 smithi157 ceph-mon[33359]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-03-11T20:49:14.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:14 smithi157 ceph-mon[33359]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T20:49:14.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:14 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:49:14.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:14 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:49:15.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:15 smithi129 ceph-mon[42066]: purged_snaps scrub starts 2024-03-11T20:49:15.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:15 smithi129 ceph-mon[42066]: purged_snaps scrub ok 2024-03-11T20:49:15.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:15 smithi129 ceph-mon[42066]: osd.3 [v2:172.21.15.129:6826/2265856698,v1:172.21.15.129:6827/2265856698] boot 2024-03-11T20:49:15.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:15 smithi129 ceph-mon[42066]: osdmap e25: 4 total, 4 up, 4 in 2024-03-11T20:49:15.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:15 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:49:15.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:15 smithi129 ceph-mon[30184]: purged_snaps scrub starts 2024-03-11T20:49:15.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:15 smithi129 ceph-mon[30184]: purged_snaps scrub ok 2024-03-11T20:49:15.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:15 smithi129 ceph-mon[30184]: osd.3 [v2:172.21.15.129:6826/2265856698,v1:172.21.15.129:6827/2265856698] boot 2024-03-11T20:49:15.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:15 smithi129 ceph-mon[30184]: osdmap e25: 4 total, 4 up, 4 in 2024-03-11T20:49:15.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:15 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:49:15.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:15 smithi157 ceph-mon[33359]: purged_snaps scrub starts 2024-03-11T20:49:15.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:15 smithi157 ceph-mon[33359]: purged_snaps scrub ok 2024-03-11T20:49:15.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:15 smithi157 ceph-mon[33359]: osd.3 [v2:172.21.15.129:6826/2265856698,v1:172.21.15.129:6827/2265856698] boot 2024-03-11T20:49:15.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:15 smithi157 ceph-mon[33359]: osdmap e25: 4 total, 4 up, 4 in 2024-03-11T20:49:15.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:15 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:49:16.151 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T20:49:16.151 INFO:teuthology.orchestra.run.smithi157.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T20:49:16.151 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10+0 records in 2024-03-11T20:49:16.151 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records out 2024-03-11T20:49:16.151 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00855015 s, 1.2 GB/s 2024-03-11T20:49:16.151 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping successful for: 2024-03-11T20:49:16.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:16 smithi157 ceph-mon[33359]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:49:16.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:16 smithi157 ceph-mon[33359]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T20:49:16.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:16 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:16.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:16 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:16.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:16 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:16.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:16 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:16.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:16 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:16.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:16 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:16.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:16 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:16.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:16 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:16.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:16 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:16.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:16 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:16.790 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi157:vg_nvme/lv_4 2024-03-11T20:49:16.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[42066]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:49:16.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[42066]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T20:49:16.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:16.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:16.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:16.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:16.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:16.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:16.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:16.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:16.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:16.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:16.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[30184]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-11T20:49:16.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[30184]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T20:49:16.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:16.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:16.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:16.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:16.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:16.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:16.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:16.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:16.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:16.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:16 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:17.023 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:49:17.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:17 smithi129 ceph-mon[30184]: Detected new or changed devices on smithi129 2024-03-11T20:49:17.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:17 smithi129 ceph-mon[30184]: Adjusting osd_memory_target on smithi129 to 4016M 2024-03-11T20:49:17.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:17 smithi129 ceph-mon[42066]: Detected new or changed devices on smithi129 2024-03-11T20:49:17.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:17 smithi129 ceph-mon[42066]: Adjusting osd_memory_target on smithi129 to 4016M 2024-03-11T20:49:17.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:17 smithi157 ceph-mon[33359]: Detected new or changed devices on smithi129 2024-03-11T20:49:17.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:17 smithi157 ceph-mon[33359]: Adjusting osd_memory_target on smithi129 to 4016M 2024-03-11T20:49:18.644 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.647+0000 7ff4da957700 1 -- 172.21.15.157:0/2325592577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4d4101e10 msgr2=0x7ff4d4102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:49:18.680 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.647+0000 7ff4da957700 1 --2- 172.21.15.157:0/2325592577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4d4101e10 0x7ff4d4102230 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009990 tx=0x7ff4c4009ca0 comp rx=0 tx=0).stop 2024-03-11T20:49:18.680 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.647+0000 7ff4da957700 1 -- 172.21.15.157:0/2325592577 shutdown_connections 2024-03-11T20:49:18.680 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.647+0000 7ff4da957700 1 --2- 172.21.15.157:0/2325592577 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff4d4103930 0x7ff4d410a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:18.680 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.647+0000 7ff4da957700 1 --2- 172.21.15.157:0/2325592577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4d4101e10 0x7ff4d4102230 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:18.680 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.647+0000 7ff4da957700 1 --2- 172.21.15.157:0/2325592577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4d4102f70 0x7ff4d41033f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:18.680 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.647+0000 7ff4da957700 1 -- 172.21.15.157:0/2325592577 >> 172.21.15.157:0/2325592577 conn(0x7ff4d40fd3f0 msgr2=0x7ff4d40ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:49:18.681 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.647+0000 7ff4da957700 1 -- 172.21.15.157:0/2325592577 shutdown_connections 2024-03-11T20:49:18.681 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.647+0000 7ff4da957700 1 -- 172.21.15.157:0/2325592577 wait complete. 2024-03-11T20:49:18.681 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4da957700 1 Processor -- start 2024-03-11T20:49:18.681 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4da957700 1 -- start start 2024-03-11T20:49:18.681 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4da957700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff4d4101e10 0x7ff4d41974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:49:18.681 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4da957700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4d4102f70 0x7ff4d4197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:49:18.681 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4da957700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4d4103930 0x7ff4d419ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:49:18.681 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4da957700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d410ce60 con 0x7ff4d4102f70 2024-03-11T20:49:18.681 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4da957700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d410cce0 con 0x7ff4d4103930 2024-03-11T20:49:18.681 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4da957700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d410cfe0 con 0x7ff4d4101e10 2024-03-11T20:49:18.681 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4d8ef4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4d4103930 0x7ff4d419ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:49:18.681 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4d8ef4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4d4103930 0x7ff4d419ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:44840/0 (socket says 172.21.15.157:44840) 2024-03-11T20:49:18.681 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4d8ef4700 1 -- 172.21.15.157:0/2467942690 learned_addr learned my addr 172.21.15.157:0/2467942690 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:49:18.681 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4d37fe700 1 --2- 172.21.15.157:0/2467942690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4d4102f70 0x7ff4d4197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:49:18.682 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4d3fff700 1 --2- 172.21.15.157:0/2467942690 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff4d4101e10 0x7ff4d41974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:49:18.682 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4d8ef4700 1 -- 172.21.15.157:0/2467942690 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff4d4101e10 msgr2=0x7ff4d41974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:49:18.682 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4d8ef4700 1 --2- 172.21.15.157:0/2467942690 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff4d4101e10 0x7ff4d41974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:18.682 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4d8ef4700 1 -- 172.21.15.157:0/2467942690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4d4102f70 msgr2=0x7ff4d4197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:49:18.682 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4d8ef4700 1 --2- 172.21.15.157:0/2467942690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4d4102f70 0x7ff4d4197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:18.682 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.648+0000 7ff4d8ef4700 1 -- 172.21.15.157:0/2467942690 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d419f0d0 con 0x7ff4d4103930 2024-03-11T20:49:18.682 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.649+0000 7ff4d8ef4700 1 --2- 172.21.15.157:0/2467942690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4d4103930 0x7ff4d419ea30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc007d70 tx=0x7ff4cc00a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:49:18.682 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.649+0000 7ff4d17fa700 1 -- 172.21.15.157:0/2467942690 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc016020 con 0x7ff4d4103930 2024-03-11T20:49:18.682 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.649+0000 7ff4d17fa700 1 -- 172.21.15.157:0/2467942690 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ff4cc00ac90 con 0x7ff4d4103930 2024-03-11T20:49:18.682 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.649+0000 7ff4da957700 1 -- 172.21.15.157:0/2467942690 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d419f3c0 con 0x7ff4d4103930 2024-03-11T20:49:18.682 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.649+0000 7ff4d17fa700 1 -- 172.21.15.157:0/2467942690 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4cc011380 con 0x7ff4d4103930 2024-03-11T20:49:18.682 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.649+0000 7ff4da957700 1 -- 172.21.15.157:0/2467942690 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d419f950 con 0x7ff4d4103930 2024-03-11T20:49:18.683 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.650+0000 7ff4da957700 1 -- 172.21.15.157:0/2467942690 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d4068aa0 con 0x7ff4d4103930 2024-03-11T20:49:18.683 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.650+0000 7ff4d17fa700 1 -- 172.21.15.157:0/2467942690 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7ff4cc019020 con 0x7ff4d4103930 2024-03-11T20:49:18.683 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.651+0000 7ff4d17fa700 1 --2- 172.21.15.157:0/2467942690 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff4bc06dd90 0x7ff4bc070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:49:18.684 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.651+0000 7ff4d17fa700 1 -- 172.21.15.157:0/2467942690 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3449+0+0 (secure 0 0 0) 0x7ff4cc08cfc0 con 0x7ff4d4103930 2024-03-11T20:49:18.684 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.651+0000 7ff4d3fff700 1 --2- 172.21.15.157:0/2467942690 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff4bc06dd90 0x7ff4bc070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:49:18.684 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.651+0000 7ff4d3fff700 1 --2- 172.21.15.157:0/2467942690 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff4bc06dd90 0x7ff4bc070250 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff4c400b5c0 tx=0x7ff4c4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:49:18.684 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.655+0000 7ff4d17fa700 1 -- 172.21.15.157:0/2467942690 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7ff4cc020020 con 0x7ff4d4103930 2024-03-11T20:49:18.858 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:18.858+0000 7ff4da957700 1 -- 172.21.15.157:0/2467942690 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4d4062220 con 0x7ff4bc06dd90 2024-03-11T20:49:18.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:18 smithi129 ceph-mon[30184]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:18.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:18 smithi129 ceph-mon[42066]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:18.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:18 smithi157 ceph-mon[33359]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:19.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:19 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:49:19.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:19 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:49:19.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:19 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:19.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:19 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:49:19.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:19 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:49:19.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:19 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:19.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:19 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:49:19.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:19 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:49:19.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:19 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:20.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:20 smithi129 ceph-mon[42066]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:49:20.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:20 smithi129 ceph-mon[42066]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:20.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:20 smithi129 ceph-mon[30184]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:49:20.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:20 smithi129 ceph-mon[30184]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:20.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:20 smithi157 ceph-mon[33359]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:49:20.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:20 smithi157 ceph-mon[33359]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:21.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.157:0/2131223060' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa92e2f-5687-4260-985a-3006da3cc803"}]: dispatch 2024-03-11T20:49:21.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa92e2f-5687-4260-985a-3006da3cc803"}]: dispatch 2024-03-11T20:49:21.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aa92e2f-5687-4260-985a-3006da3cc803"}]': finished 2024-03-11T20:49:21.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:21 smithi129 ceph-mon[30184]: osdmap e27: 5 total, 4 up, 5 in 2024-03-11T20:49:21.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:21 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:49:21.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.157:0/2131223060' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa92e2f-5687-4260-985a-3006da3cc803"}]: dispatch 2024-03-11T20:49:21.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa92e2f-5687-4260-985a-3006da3cc803"}]: dispatch 2024-03-11T20:49:21.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aa92e2f-5687-4260-985a-3006da3cc803"}]': finished 2024-03-11T20:49:21.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:21 smithi129 ceph-mon[42066]: osdmap e27: 5 total, 4 up, 5 in 2024-03-11T20:49:21.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:21 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:49:21.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.157:0/2131223060' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa92e2f-5687-4260-985a-3006da3cc803"}]: dispatch 2024-03-11T20:49:21.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2aa92e2f-5687-4260-985a-3006da3cc803"}]: dispatch 2024-03-11T20:49:21.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2aa92e2f-5687-4260-985a-3006da3cc803"}]': finished 2024-03-11T20:49:21.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:21 smithi157 ceph-mon[33359]: osdmap e27: 5 total, 4 up, 5 in 2024-03-11T20:49:21.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:21 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:49:22.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:22 smithi129 ceph-mon[42066]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:22.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.157:0/874982422' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:49:22.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:22 smithi129 ceph-mon[30184]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:22.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.157:0/874982422' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:49:22.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:22 smithi157 ceph-mon[33359]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:22.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.157:0/874982422' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:49:24.969 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:24 smithi157 ceph-mon[33359]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:24.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:24 smithi129 ceph-mon[30184]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:24.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:24 smithi129 ceph-mon[42066]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:26.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:26 smithi129 ceph-mon[30184]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:26.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:26 smithi129 ceph-mon[42066]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:26.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:26 smithi157 ceph-mon[33359]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:28.769 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:28 smithi157 ceph-mon[33359]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:28.769 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:28 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T20:49:28.769 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:28 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:28.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:28 smithi129 ceph-mon[30184]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:28.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:28 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T20:49:28.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:28 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:28.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:28 smithi129 ceph-mon[42066]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:28.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:28 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T20:49:28.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:28 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:29.842 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:29 smithi157 ceph-mon[33359]: Deploying daemon osd.4 on smithi157 2024-03-11T20:49:29.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:29 smithi129 ceph-mon[30184]: Deploying daemon osd.4 on smithi157 2024-03-11T20:49:29.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:29 smithi129 ceph-mon[42066]: Deploying daemon osd.4 on smithi157 2024-03-11T20:49:30.779 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:30 smithi157 ceph-mon[33359]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:30.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:30 smithi129 ceph-mon[30184]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:30.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:30 smithi129 ceph-mon[42066]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:32.863 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:32 smithi157 ceph-mon[33359]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:32.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:32 smithi129 ceph-mon[30184]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:32.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:32 smithi129 ceph-mon[42066]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:34.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:34 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:34.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:34 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:34.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:34 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:34.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:34 smithi129 ceph-mon[30184]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:34.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:34 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:34.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:34 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:34.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:34 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:34.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:34 smithi129 ceph-mon[42066]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:34.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:34 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:34.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:34 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:34.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:34 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:34.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:34 smithi157 ceph-mon[33359]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:36.807 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:36 smithi157 ceph-mon[33359]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:36.807 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:36 smithi157 ceph-mon[33359]: from='osd.4 [v2:172.21.15.157:6800/3991533346,v1:172.21.15.157:6801/3991533346]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T20:49:36.807 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:36 smithi157 ceph-mon[33359]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T20:49:36.866 INFO:teuthology.orchestra.run.smithi157.stdout:Created osd(s) 4 on host 'smithi157' 2024-03-11T20:49:36.866 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:36.870+0000 7ff4d17fa700 1 -- 172.21.15.157:0/2467942690 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff4d4062220 con 0x7ff4bc06dd90 2024-03-11T20:49:36.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:36.872+0000 7ff4da957700 1 -- 172.21.15.157:0/2467942690 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff4bc06dd90 msgr2=0x7ff4bc070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:49:36.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:36.872+0000 7ff4da957700 1 --2- 172.21.15.157:0/2467942690 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff4bc06dd90 0x7ff4bc070250 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff4c400b5c0 tx=0x7ff4c4005960 comp rx=0 tx=0).stop 2024-03-11T20:49:36.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:36.872+0000 7ff4da957700 1 -- 172.21.15.157:0/2467942690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4d4103930 msgr2=0x7ff4d419ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:49:36.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:36.872+0000 7ff4da957700 1 --2- 172.21.15.157:0/2467942690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4d4103930 0x7ff4d419ea30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc007d70 tx=0x7ff4cc00a340 comp rx=0 tx=0).stop 2024-03-11T20:49:36.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:36.872+0000 7ff4da957700 1 -- 172.21.15.157:0/2467942690 shutdown_connections 2024-03-11T20:49:36.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:36.872+0000 7ff4da957700 1 --2- 172.21.15.157:0/2467942690 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff4bc06dd90 0x7ff4bc070250 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:36.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:36.872+0000 7ff4da957700 1 --2- 172.21.15.157:0/2467942690 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4d4103930 0x7ff4d419ea30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:36.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:36.872+0000 7ff4da957700 1 --2- 172.21.15.157:0/2467942690 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff4d4101e10 0x7ff4d41974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:36.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:36.872+0000 7ff4da957700 1 --2- 172.21.15.157:0/2467942690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4d4102f70 0x7ff4d4197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:36.869 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:36.872+0000 7ff4da957700 1 -- 172.21.15.157:0/2467942690 >> 172.21.15.157:0/2467942690 conn(0x7ff4d40fd3f0 msgr2=0x7ff4d40feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:49:36.869 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:36.872+0000 7ff4da957700 1 -- 172.21.15.157:0/2467942690 shutdown_connections 2024-03-11T20:49:36.869 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:36.872+0000 7ff4da957700 1 -- 172.21.15.157:0/2467942690 wait complete. 2024-03-11T20:49:36.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:36 smithi129 ceph-mon[30184]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:36.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:36 smithi129 ceph-mon[30184]: from='osd.4 [v2:172.21.15.157:6800/3991533346,v1:172.21.15.157:6801/3991533346]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T20:49:36.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:36 smithi129 ceph-mon[30184]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T20:49:36.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:36 smithi129 ceph-mon[42066]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:36.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:36 smithi129 ceph-mon[42066]: from='osd.4 [v2:172.21.15.157:6800/3991533346,v1:172.21.15.157:6801/3991533346]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T20:49:36.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:36 smithi129 ceph-mon[42066]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T20:49:37.479 DEBUG:teuthology.orchestra.run.smithi157:osd.4> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.4.service 2024-03-11T20:49:37.482 INFO:tasks.cephadm:Deploying osd.5 on smithi157 with /dev/vg_nvme/lv_3... 2024-03-11T20:49:37.482 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T20:49:37.721 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:37 smithi157 ceph-mon[33359]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T20:49:37.721 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:37 smithi157 ceph-mon[33359]: from='osd.4 [v2:172.21.15.157:6800/3991533346,v1:172.21.15.157:6801/3991533346]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:49:37.721 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:37 smithi157 ceph-mon[33359]: osdmap e28: 5 total, 4 up, 5 in 2024-03-11T20:49:37.721 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:37 smithi157 ceph-mon[33359]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:49:37.721 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:37 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:49:37.721 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:37 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:37.721 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:37 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:37.721 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:37 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:37.721 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:37 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:37.722 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:37 smithi157 ceph-mon[33359]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-03-11T20:49:37.722 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:37 smithi157 ceph-mon[33359]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T20:49:37.722 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 20:49:37 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T20:49:37.665+0000 7fa0995ce700 -1 osd.4 0 waiting for initial osdmap 2024-03-11T20:49:37.722 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 20:49:37 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T20:49:37.672+0000 7fa0953c1700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T20:49:37.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[30184]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T20:49:37.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[30184]: from='osd.4 [v2:172.21.15.157:6800/3991533346,v1:172.21.15.157:6801/3991533346]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:49:37.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[30184]: osdmap e28: 5 total, 4 up, 5 in 2024-03-11T20:49:37.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[30184]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:49:37.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:49:37.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:37.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[30184]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-03-11T20:49:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[30184]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T20:49:37.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[42066]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T20:49:37.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[42066]: from='osd.4 [v2:172.21.15.157:6800/3991533346,v1:172.21.15.157:6801/3991533346]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:49:37.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[42066]: osdmap e28: 5 total, 4 up, 5 in 2024-03-11T20:49:37.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[42066]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:49:37.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:49:37.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:37.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:37.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:37.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:37.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[42066]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-03-11T20:49:37.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:37 smithi129 ceph-mon[42066]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T20:49:38.262 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:49:38.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:38 smithi129 ceph-mon[30184]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:38.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:38 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:49:38.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:38 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:49:38.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:38 smithi129 ceph-mon[42066]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:38.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:38 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:49:38.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:38 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:49:38.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:38 smithi157 ceph-mon[33359]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T20:49:38.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:38 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:49:38.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:38 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:49:39.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:39 smithi157 ceph-mon[33359]: purged_snaps scrub starts 2024-03-11T20:49:39.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:39 smithi157 ceph-mon[33359]: purged_snaps scrub ok 2024-03-11T20:49:39.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:39 smithi157 ceph-mon[33359]: osd.4 [v2:172.21.15.157:6800/3991533346,v1:172.21.15.157:6801/3991533346] boot 2024-03-11T20:49:39.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:39 smithi157 ceph-mon[33359]: osdmap e30: 5 total, 5 up, 5 in 2024-03-11T20:49:39.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:39 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:49:39.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:39 smithi157 ceph-mon[33359]: osdmap e31: 5 total, 5 up, 5 in 2024-03-11T20:49:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:39 smithi129 ceph-mon[30184]: purged_snaps scrub starts 2024-03-11T20:49:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:39 smithi129 ceph-mon[30184]: purged_snaps scrub ok 2024-03-11T20:49:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:39 smithi129 ceph-mon[30184]: osd.4 [v2:172.21.15.157:6800/3991533346,v1:172.21.15.157:6801/3991533346] boot 2024-03-11T20:49:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:39 smithi129 ceph-mon[30184]: osdmap e30: 5 total, 5 up, 5 in 2024-03-11T20:49:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:39 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:49:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:39 smithi129 ceph-mon[30184]: osdmap e31: 5 total, 5 up, 5 in 2024-03-11T20:49:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:39 smithi129 ceph-mon[42066]: purged_snaps scrub starts 2024-03-11T20:49:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:39 smithi129 ceph-mon[42066]: purged_snaps scrub ok 2024-03-11T20:49:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:39 smithi129 ceph-mon[42066]: osd.4 [v2:172.21.15.157:6800/3991533346,v1:172.21.15.157:6801/3991533346] boot 2024-03-11T20:49:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:39 smithi129 ceph-mon[42066]: osdmap e30: 5 total, 5 up, 5 in 2024-03-11T20:49:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:39 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:49:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:39 smithi129 ceph-mon[42066]: osdmap e31: 5 total, 5 up, 5 in 2024-03-11T20:49:40.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:40 smithi129 ceph-mon[30184]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:49:40.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:40 smithi129 ceph-mon[42066]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:49:40.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:40 smithi157 ceph-mon[33359]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:49:41.052 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T20:49:41.052 INFO:teuthology.orchestra.run.smithi157.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T20:49:41.053 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10+0 records in 2024-03-11T20:49:41.053 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records out 2024-03-11T20:49:41.053 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0107136 s, 979 MB/s 2024-03-11T20:49:41.053 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping successful for: 2024-03-11T20:49:41.863 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:41 smithi157 ceph-mon[33359]: osdmap e32: 5 total, 5 up, 5 in 2024-03-11T20:49:41.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:41 smithi129 ceph-mon[30184]: osdmap e32: 5 total, 5 up, 5 in 2024-03-11T20:49:41.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:41 smithi129 ceph-mon[42066]: osdmap e32: 5 total, 5 up, 5 in 2024-03-11T20:49:42.316 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi157:vg_nvme/lv_3 2024-03-11T20:49:42.562 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:49:42.952 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:42 smithi157 ceph-mon[33359]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:49:42.952 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:42 smithi157 ceph-mon[33359]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T20:49:42.952 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:42 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:42.952 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:42 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:42.952 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:42 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:42.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:42 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:42.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:42 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:42.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:42 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:42.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:42 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:42.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:42 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:42.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:42 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:42.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:42 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:42.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:42 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:42.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[30184]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:49:42.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[30184]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T20:49:42.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:42.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:42.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[42066]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:49:42.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[42066]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T20:49:42.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:42.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:42.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:49:42.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:42.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:42.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:42.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:42.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:49:42.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:42.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:49:42.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:42 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:49:43.691 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:43 smithi129 ceph-mon[30184]: Detected new or changed devices on smithi157 2024-03-11T20:49:43.691 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:43 smithi129 ceph-mon[30184]: Adjusting osd_memory_target on smithi157 to 17088M 2024-03-11T20:49:43.691 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:43 smithi129 ceph-mon[30184]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T20:49:43.691 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:43 smithi129 ceph-mon[42066]: Detected new or changed devices on smithi157 2024-03-11T20:49:43.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:43 smithi129 ceph-mon[42066]: Adjusting osd_memory_target on smithi157 to 17088M 2024-03-11T20:49:43.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:43 smithi129 ceph-mon[42066]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T20:49:43.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:43 smithi157 ceph-mon[33359]: Detected new or changed devices on smithi157 2024-03-11T20:49:43.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:43 smithi157 ceph-mon[33359]: Adjusting osd_memory_target on smithi157 to 17088M 2024-03-11T20:49:43.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:43 smithi157 ceph-mon[33359]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T20:49:44.205 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.210+0000 7fc82fe92700 1 -- 172.21.15.157:0/2298314428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc828107bb0 msgr2=0x7fc828109fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:49:44.206 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.210+0000 7fc82fe92700 1 --2- 172.21.15.157:0/2298314428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc828107bb0 0x7fc828109fa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc82000b0a0 tx=0x7fc82000b3b0 comp rx=0 tx=0).stop 2024-03-11T20:49:44.206 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.210+0000 7fc82fe92700 1 -- 172.21.15.157:0/2298314428 shutdown_connections 2024-03-11T20:49:44.206 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.210+0000 7fc82fe92700 1 --2- 172.21.15.157:0/2298314428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc828107bb0 0x7fc828109fa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:44.206 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.210+0000 7fc82fe92700 1 --2- 172.21.15.157:0/2298314428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc828102950 0x7fc828104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:44.206 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.210+0000 7fc82fe92700 1 --2- 172.21.15.157:0/2298314428 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc828105280 0x7fc828107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:44.206 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.210+0000 7fc82fe92700 1 -- 172.21.15.157:0/2298314428 >> 172.21.15.157:0/2298314428 conn(0x7fc8280fc300 msgr2=0x7fc8280fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:49:44.206 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.210+0000 7fc82fe92700 1 -- 172.21.15.157:0/2298314428 shutdown_connections 2024-03-11T20:49:44.206 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.210+0000 7fc82fe92700 1 -- 172.21.15.157:0/2298314428 wait complete. 2024-03-11T20:49:44.208 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.210+0000 7fc82fe92700 1 Processor -- start 2024-03-11T20:49:44.208 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82fe92700 1 -- start start 2024-03-11T20:49:44.208 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82fe92700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc828102950 0x7fc828197550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:49:44.208 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82fe92700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc828105280 0x7fc828197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:49:44.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82fe92700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc828107bb0 0x7fc82819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:49:44.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82fe92700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82810ce90 con 0x7fc828107bb0 2024-03-11T20:49:44.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82fe92700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82810cd10 con 0x7fc828105280 2024-03-11T20:49:44.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82fe92700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82810d010 con 0x7fc828102950 2024-03-11T20:49:44.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82d42d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc828105280 0x7fc828197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:49:44.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82dc2e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc828102950 0x7fc828197550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:49:44.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82d42d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc828105280 0x7fc828197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:42906/0 (socket says 172.21.15.157:42906) 2024-03-11T20:49:44.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82d42d700 1 -- 172.21.15.157:0/3307884305 learned_addr learned my addr 172.21.15.157:0/3307884305 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:49:44.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82e42f700 1 --2- 172.21.15.157:0/3307884305 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc828107bb0 0x7fc82819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:49:44.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82d42d700 1 -- 172.21.15.157:0/3307884305 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc828102950 msgr2=0x7fc828197550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:49:44.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82d42d700 1 --2- 172.21.15.157:0/3307884305 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc828102950 0x7fc828197550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:44.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82d42d700 1 -- 172.21.15.157:0/3307884305 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc828107bb0 msgr2=0x7fc82819eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:49:44.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82d42d700 1 --2- 172.21.15.157:0/3307884305 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc828107bb0 0x7fc82819eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:49:44.209 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82d42d700 1 -- 172.21.15.157:0/3307884305 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82819f160 con 0x7fc828105280 2024-03-11T20:49:44.210 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc82d42d700 1 --2- 172.21.15.157:0/3307884305 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc828105280 0x7fc828197a90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc81800d980 tx=0x7fc81800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:49:44.210 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.211+0000 7fc81effd700 1 -- 172.21.15.157:0/3307884305 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc818017070 con 0x7fc828105280 2024-03-11T20:49:44.210 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.212+0000 7fc81effd700 1 -- 172.21.15.157:0/3307884305 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc818005020 con 0x7fc828105280 2024-03-11T20:49:44.210 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.212+0000 7fc81effd700 1 -- 172.21.15.157:0/3307884305 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81800f970 con 0x7fc828105280 2024-03-11T20:49:44.210 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.212+0000 7fc82fe92700 1 -- 172.21.15.157:0/3307884305 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc82819f450 con 0x7fc828105280 2024-03-11T20:49:44.210 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.212+0000 7fc82fe92700 1 -- 172.21.15.157:0/3307884305 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82819f940 con 0x7fc828105280 2024-03-11T20:49:44.210 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.213+0000 7fc82fe92700 1 -- 172.21.15.157:0/3307884305 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc828068aa0 con 0x7fc828105280 2024-03-11T20:49:44.210 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.213+0000 7fc81effd700 1 -- 172.21.15.157:0/3307884305 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7fc818004a30 con 0x7fc828105280 2024-03-11T20:49:44.211 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.213+0000 7fc81effd700 1 --2- 172.21.15.157:0/3307884305 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fc81406dd90 0x7fc814070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:49:44.211 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.214+0000 7fc81effd700 1 -- 172.21.15.157:0/3307884305 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 3991+0+0 (secure 0 0 0) 0x7fc81808d0e0 con 0x7fc828105280 2024-03-11T20:49:44.211 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.214+0000 7fc82dc2e700 1 --2- 172.21.15.157:0/3307884305 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fc81406dd90 0x7fc814070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:49:44.211 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.214+0000 7fc82dc2e700 1 --2- 172.21.15.157:0/3307884305 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fc81406dd90 0x7fc814070250 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc824005950 tx=0x7fc82400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:49:44.214 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.218+0000 7fc81effd700 1 -- 172.21.15.157:0/3307884305 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fc818020020 con 0x7fc828105280 2024-03-11T20:49:44.417 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:49:44.421+0000 7fc82fe92700 1 -- 172.21.15.157:0/3307884305 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc828062220 con 0x7fc81406dd90 2024-03-11T20:49:44.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:44 smithi129 ceph-mon[42066]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:49:44.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:44 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:49:44.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:44 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:49:44.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:44 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:44.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:44 smithi129 ceph-mon[30184]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:49:44.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:44 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:49:44.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:44 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:49:44.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:44 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:44.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:44 smithi157 ceph-mon[33359]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:49:44.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:44 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:49:44.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:44 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:49:44.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:44 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:45.955 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:45 smithi157 ceph-mon[33359]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:49:45.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:45 smithi129 ceph-mon[30184]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:49:45.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:45 smithi129 ceph-mon[42066]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:49:46.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:46 smithi129 ceph-mon[30184]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T20:49:46.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:46 smithi129 ceph-mon[42066]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T20:49:46.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:46 smithi157 ceph-mon[33359]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T20:49:47.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.157:0/2509716529' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dac5fa60-f01b-4323-8596-cc0f3a5030c1"}]: dispatch 2024-03-11T20:49:47.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:47 smithi129 ceph-mon[30184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dac5fa60-f01b-4323-8596-cc0f3a5030c1"}]: dispatch 2024-03-11T20:49:47.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:47 smithi129 ceph-mon[30184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dac5fa60-f01b-4323-8596-cc0f3a5030c1"}]': finished 2024-03-11T20:49:47.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:47 smithi129 ceph-mon[30184]: osdmap e35: 6 total, 5 up, 6 in 2024-03-11T20:49:47.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:47 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:49:47.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.157:0/2337469956' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:49:47.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.157:0/2509716529' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dac5fa60-f01b-4323-8596-cc0f3a5030c1"}]: dispatch 2024-03-11T20:49:47.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:47 smithi129 ceph-mon[42066]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dac5fa60-f01b-4323-8596-cc0f3a5030c1"}]: dispatch 2024-03-11T20:49:47.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:47 smithi129 ceph-mon[42066]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dac5fa60-f01b-4323-8596-cc0f3a5030c1"}]': finished 2024-03-11T20:49:47.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:47 smithi129 ceph-mon[42066]: osdmap e35: 6 total, 5 up, 6 in 2024-03-11T20:49:47.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:47 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:49:47.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.157:0/2337469956' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:49:47.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.157:0/2509716529' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dac5fa60-f01b-4323-8596-cc0f3a5030c1"}]: dispatch 2024-03-11T20:49:47.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:47 smithi157 ceph-mon[33359]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dac5fa60-f01b-4323-8596-cc0f3a5030c1"}]: dispatch 2024-03-11T20:49:47.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:47 smithi157 ceph-mon[33359]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dac5fa60-f01b-4323-8596-cc0f3a5030c1"}]': finished 2024-03-11T20:49:47.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:47 smithi157 ceph-mon[33359]: osdmap e35: 6 total, 5 up, 6 in 2024-03-11T20:49:47.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:47 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:49:47.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.157:0/2337469956' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:49:48.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:48 smithi129 ceph-mon[30184]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T20:49:48.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:48 smithi129 ceph-mon[42066]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T20:49:48.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:48 smithi157 ceph-mon[33359]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T20:49:50.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:50 smithi157 ceph-mon[33359]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T20:49:50.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:50 smithi129 ceph-mon[30184]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T20:49:50.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:50 smithi129 ceph-mon[42066]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-11T20:49:52.840 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:52 smithi157 ceph-mon[33359]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T20:49:52.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:52 smithi129 ceph-mon[30184]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T20:49:52.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:52 smithi129 ceph-mon[42066]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-11T20:49:53.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:53 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T20:49:53.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:53 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:53.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:53 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T20:49:53.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:53 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:53.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:53 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T20:49:53.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:53 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:49:54.957 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:54 smithi157 ceph-mon[33359]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:49:54.957 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:54 smithi157 ceph-mon[33359]: Deploying daemon osd.5 on smithi157 2024-03-11T20:49:54.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:54 smithi129 ceph-mon[30184]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:49:54.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:54 smithi129 ceph-mon[30184]: Deploying daemon osd.5 on smithi157 2024-03-11T20:49:54.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:54 smithi129 ceph-mon[42066]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:49:54.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:54 smithi129 ceph-mon[42066]: Deploying daemon osd.5 on smithi157 2024-03-11T20:49:56.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:56 smithi129 ceph-mon[30184]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:49:56.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:56 smithi129 ceph-mon[42066]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:49:56.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:56 smithi157 ceph-mon[33359]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:49:58.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:49:58 smithi129 ceph-mon[42066]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:49:58.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:49:58 smithi129 ceph-mon[30184]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:49:58.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:49:58 smithi157 ceph-mon[33359]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:50:00.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:00 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:00.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:00 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:00.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:00 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:00.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:00 smithi129 ceph-mon[30184]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:50:00.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:00 smithi129 ceph-mon[30184]: overall HEALTH_OK 2024-03-11T20:50:00.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:00 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:00.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:00 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:00.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:00 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:00.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:00 smithi129 ceph-mon[42066]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:50:00.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:00 smithi129 ceph-mon[42066]: overall HEALTH_OK 2024-03-11T20:50:00.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:00 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:00.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:00 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:00.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:00 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:00.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:00 smithi157 ceph-mon[33359]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:50:00.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:00 smithi157 ceph-mon[33359]: overall HEALTH_OK 2024-03-11T20:50:02.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:02 smithi157 ceph-mon[33359]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:50:02.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:02 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:02.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:02 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:02.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:02 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:02.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:02 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:02.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:02 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:02.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:02 smithi157 ceph-mon[33359]: from='osd.5 [v2:172.21.15.157:6808/2444017632,v1:172.21.15.157:6809/2444017632]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T20:50:02.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:02 smithi157 ceph-mon[33359]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T20:50:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:02 smithi129 ceph-mon[30184]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:50:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:02 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:02 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:02 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:02 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:02 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:02 smithi129 ceph-mon[30184]: from='osd.5 [v2:172.21.15.157:6808/2444017632,v1:172.21.15.157:6809/2444017632]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T20:50:02.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:02 smithi129 ceph-mon[30184]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T20:50:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:02 smithi129 ceph-mon[42066]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:50:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:02 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:02 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:02 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:02 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:02 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:02 smithi129 ceph-mon[42066]: from='osd.5 [v2:172.21.15.157:6808/2444017632,v1:172.21.15.157:6809/2444017632]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T20:50:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:02 smithi129 ceph-mon[42066]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T20:50:03.130 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:03.134+0000 7fc81effd700 1 -- 172.21.15.157:0/3307884305 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc828062220 con 0x7fc81406dd90 2024-03-11T20:50:03.130 INFO:teuthology.orchestra.run.smithi157.stdout:Created osd(s) 5 on host 'smithi157' 2024-03-11T20:50:03.132 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:03.136+0000 7fc82fe92700 1 -- 172.21.15.157:0/3307884305 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fc81406dd90 msgr2=0x7fc814070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:03.132 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:03.136+0000 7fc82fe92700 1 --2- 172.21.15.157:0/3307884305 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fc81406dd90 0x7fc814070250 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc824005950 tx=0x7fc82400b410 comp rx=0 tx=0).stop 2024-03-11T20:50:03.132 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:03.136+0000 7fc82fe92700 1 -- 172.21.15.157:0/3307884305 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc828105280 msgr2=0x7fc828197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:03.132 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:03.136+0000 7fc82fe92700 1 --2- 172.21.15.157:0/3307884305 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc828105280 0x7fc828197a90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc81800d980 tx=0x7fc81800dc90 comp rx=0 tx=0).stop 2024-03-11T20:50:03.132 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:03.136+0000 7fc82fe92700 1 -- 172.21.15.157:0/3307884305 shutdown_connections 2024-03-11T20:50:03.133 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:03.136+0000 7fc82fe92700 1 --2- 172.21.15.157:0/3307884305 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fc81406dd90 0x7fc814070250 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:03.133 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:03.136+0000 7fc82fe92700 1 --2- 172.21.15.157:0/3307884305 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc828107bb0 0x7fc82819eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:03.133 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:03.136+0000 7fc82fe92700 1 --2- 172.21.15.157:0/3307884305 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc828102950 0x7fc828197550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:03.133 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:03.136+0000 7fc82fe92700 1 --2- 172.21.15.157:0/3307884305 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc828105280 0x7fc828197a90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:03.133 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:03.136+0000 7fc82fe92700 1 -- 172.21.15.157:0/3307884305 >> 172.21.15.157:0/3307884305 conn(0x7fc8280fc300 msgr2=0x7fc8281035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:50:03.133 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:03.137+0000 7fc82fe92700 1 -- 172.21.15.157:0/3307884305 shutdown_connections 2024-03-11T20:50:03.133 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:03.137+0000 7fc82fe92700 1 -- 172.21.15.157:0/3307884305 wait complete. 2024-03-11T20:50:03.723 DEBUG:teuthology.orchestra.run.smithi157:osd.5> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.5.service 2024-03-11T20:50:03.726 INFO:tasks.cephadm:Deploying osd.6 on smithi157 with /dev/vg_nvme/lv_2... 2024-03-11T20:50:03.726 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T20:50:04.362 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:04 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:04.362 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:04 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:04.362 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:04 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:04.362 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:04 smithi157 ceph-mon[33359]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T20:50:04.362 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:04 smithi157 ceph-mon[33359]: from='osd.5 [v2:172.21.15.157:6808/2444017632,v1:172.21.15.157:6809/2444017632]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:04.362 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:04 smithi157 ceph-mon[33359]: osdmap e36: 6 total, 5 up, 6 in 2024-03-11T20:50:04.362 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:04 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:50:04.362 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:04 smithi157 ceph-mon[33359]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:04.362 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:04 smithi157 ceph-mon[33359]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:50:04.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:04.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:04.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:04.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[30184]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T20:50:04.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[30184]: from='osd.5 [v2:172.21.15.157:6808/2444017632,v1:172.21.15.157:6809/2444017632]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:04.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[30184]: osdmap e36: 6 total, 5 up, 6 in 2024-03-11T20:50:04.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:50:04.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[30184]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:04.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[30184]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:50:04.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:04.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:04.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:04.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[42066]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T20:50:04.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[42066]: from='osd.5 [v2:172.21.15.157:6808/2444017632,v1:172.21.15.157:6809/2444017632]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:04.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[42066]: osdmap e36: 6 total, 5 up, 6 in 2024-03-11T20:50:04.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:50:04.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[42066]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:04.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:04 smithi129 ceph-mon[42066]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T20:50:04.687 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 20:50:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T20:50:04.553+0000 7f86edff2700 -1 osd.5 0 waiting for initial osdmap 2024-03-11T20:50:04.687 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 20:50:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T20:50:04.557+0000 7f86e5ddd700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T20:50:05.016 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:50:05.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:05 smithi157 ceph-mon[33359]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-03-11T20:50:05.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:05 smithi157 ceph-mon[33359]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T20:50:05.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:05 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:50:05.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:05 smithi157 ceph-mon[33359]: from='osd.5 ' entity='osd.5' 2024-03-11T20:50:05.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:05 smithi129 ceph-mon[30184]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-03-11T20:50:05.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:05 smithi129 ceph-mon[30184]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T20:50:05.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:05 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:50:05.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:05 smithi129 ceph-mon[30184]: from='osd.5 ' entity='osd.5' 2024-03-11T20:50:05.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:05 smithi129 ceph-mon[42066]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-03-11T20:50:05.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:05 smithi129 ceph-mon[42066]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T20:50:05.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:05 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:50:05.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:05 smithi129 ceph-mon[42066]: from='osd.5 ' entity='osd.5' 2024-03-11T20:50:06.464 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:06 smithi157 ceph-mon[33359]: purged_snaps scrub starts 2024-03-11T20:50:06.464 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:06 smithi157 ceph-mon[33359]: purged_snaps scrub ok 2024-03-11T20:50:06.464 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:06 smithi157 ceph-mon[33359]: osd.5 [v2:172.21.15.157:6808/2444017632,v1:172.21.15.157:6809/2444017632] boot 2024-03-11T20:50:06.464 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:06 smithi157 ceph-mon[33359]: osdmap e38: 6 total, 6 up, 6 in 2024-03-11T20:50:06.464 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:06 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:50:06.464 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:06 smithi157 ceph-mon[33359]: osdmap e39: 6 total, 6 up, 6 in 2024-03-11T20:50:06.464 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:06 smithi157 ceph-mon[33359]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:06.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:06 smithi129 ceph-mon[30184]: purged_snaps scrub starts 2024-03-11T20:50:06.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:06 smithi129 ceph-mon[30184]: purged_snaps scrub ok 2024-03-11T20:50:06.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:06 smithi129 ceph-mon[30184]: osd.5 [v2:172.21.15.157:6808/2444017632,v1:172.21.15.157:6809/2444017632] boot 2024-03-11T20:50:06.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:06 smithi129 ceph-mon[30184]: osdmap e38: 6 total, 6 up, 6 in 2024-03-11T20:50:06.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:06 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:50:06.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:06 smithi129 ceph-mon[30184]: osdmap e39: 6 total, 6 up, 6 in 2024-03-11T20:50:06.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:06 smithi129 ceph-mon[30184]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:06.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:06 smithi129 ceph-mon[42066]: purged_snaps scrub starts 2024-03-11T20:50:06.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:06 smithi129 ceph-mon[42066]: purged_snaps scrub ok 2024-03-11T20:50:06.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:06 smithi129 ceph-mon[42066]: osd.5 [v2:172.21.15.157:6808/2444017632,v1:172.21.15.157:6809/2444017632] boot 2024-03-11T20:50:06.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:06 smithi129 ceph-mon[42066]: osdmap e38: 6 total, 6 up, 6 in 2024-03-11T20:50:06.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:06 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:50:06.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:06 smithi129 ceph-mon[42066]: osdmap e39: 6 total, 6 up, 6 in 2024-03-11T20:50:06.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:06 smithi129 ceph-mon[42066]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:07.630 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:07 smithi157 ceph-mon[33359]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T20:50:07.630 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:07 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:07.630 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:07 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:07.631 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:07 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:07.631 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:07 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:07.631 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:07 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:07.631 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:07 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:07.631 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:07 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:07.631 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:07 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:07.973 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T20:50:07.973 INFO:teuthology.orchestra.run.smithi157.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T20:50:07.973 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10+0 records in 2024-03-11T20:50:07.974 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records out 2024-03-11T20:50:07.974 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00854682 s, 1.2 GB/s 2024-03-11T20:50:07.974 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping successful for: 2024-03-11T20:50:07.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[30184]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T20:50:07.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:07.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:07.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:07.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:07.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:07.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:07.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:07.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:07.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[42066]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T20:50:07.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:07.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:07.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:07.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:07.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:07.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:07.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:07.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:07 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:08.614 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi157:vg_nvme/lv_2 2024-03-11T20:50:08.855 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:50:08.976 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:08 smithi157 ceph-mon[33359]: Detected new or changed devices on smithi157 2024-03-11T20:50:08.976 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:08 smithi157 ceph-mon[33359]: Adjusting osd_memory_target on smithi157 to 8544M 2024-03-11T20:50:08.976 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:08 smithi157 ceph-mon[33359]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:08.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:08 smithi157 ceph-mon[33359]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T20:50:08.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:08 smithi129 ceph-mon[30184]: Detected new or changed devices on smithi157 2024-03-11T20:50:08.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:08 smithi129 ceph-mon[30184]: Adjusting osd_memory_target on smithi157 to 8544M 2024-03-11T20:50:08.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:08 smithi129 ceph-mon[30184]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:08.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:08 smithi129 ceph-mon[30184]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T20:50:08.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:08 smithi129 ceph-mon[42066]: Detected new or changed devices on smithi157 2024-03-11T20:50:08.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:08 smithi129 ceph-mon[42066]: Adjusting osd_memory_target on smithi157 to 8544M 2024-03-11T20:50:08.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:08 smithi129 ceph-mon[42066]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:08.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:08 smithi129 ceph-mon[42066]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T20:50:10.443 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.445+0000 7f4dba21b700 1 -- 172.21.15.157:0/2069553053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4db4103930 msgr2=0x7f4db410a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:10.443 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.445+0000 7f4dba21b700 1 --2- 172.21.15.157:0/2069553053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4db4103930 0x7f4db410a250 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00b0a0 tx=0x7f4dac00b3b0 comp rx=0 tx=0).stop 2024-03-11T20:50:10.443 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.445+0000 7f4dba21b700 1 -- 172.21.15.157:0/2069553053 shutdown_connections 2024-03-11T20:50:10.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.445+0000 7f4dba21b700 1 --2- 172.21.15.157:0/2069553053 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4db4103930 0x7f4db410a250 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:10.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.445+0000 7f4dba21b700 1 --2- 172.21.15.157:0/2069553053 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4db4101e10 0x7f4db4102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:10.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.445+0000 7f4dba21b700 1 --2- 172.21.15.157:0/2069553053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4db4102f70 0x7f4db41033f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:10.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.445+0000 7f4dba21b700 1 -- 172.21.15.157:0/2069553053 >> 172.21.15.157:0/2069553053 conn(0x7f4db40fd3f0 msgr2=0x7f4db40ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:50:10.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.445+0000 7f4dba21b700 1 -- 172.21.15.157:0/2069553053 shutdown_connections 2024-03-11T20:50:10.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.445+0000 7f4dba21b700 1 -- 172.21.15.157:0/2069553053 wait complete. 2024-03-11T20:50:10.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4dba21b700 1 Processor -- start 2024-03-11T20:50:10.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4dba21b700 1 -- start start 2024-03-11T20:50:10.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4dba21b700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4db4101e10 0x7f4db4075b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:50:10.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4dba21b700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4db4102f70 0x7f4db40741d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:50:10.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4dba21b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4db4103930 0x7f4db4074710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:50:10.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4dba21b700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db410ce80 con 0x7f4db4101e10 2024-03-11T20:50:10.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4dba21b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db410cd00 con 0x7f4db4103930 2024-03-11T20:50:10.444 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4dba21b700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db410d000 con 0x7f4db4102f70 2024-03-11T20:50:10.445 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4db3fff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4db4103930 0x7f4db4074710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:50:10.445 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4db3fff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4db4103930 0x7f4db4074710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:59458/0 (socket says 172.21.15.157:59458) 2024-03-11T20:50:10.445 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4db3fff700 1 -- 172.21.15.157:0/1789509219 learned_addr learned my addr 172.21.15.157:0/1789509219 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:50:10.445 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4dabfff700 1 --2- 172.21.15.157:0/1789509219 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4db4102f70 0x7f4db40741d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:50:10.445 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4db37fe700 1 --2- 172.21.15.157:0/1789509219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4db4101e10 0x7f4db4075b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:50:10.445 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4db3fff700 1 -- 172.21.15.157:0/1789509219 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4db4102f70 msgr2=0x7f4db40741d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:10.445 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4db3fff700 1 --2- 172.21.15.157:0/1789509219 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4db4102f70 0x7f4db40741d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:10.445 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4db3fff700 1 -- 172.21.15.157:0/1789509219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4db4101e10 msgr2=0x7f4db4075b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:10.445 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4db3fff700 1 --2- 172.21.15.157:0/1789509219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4db4101e10 0x7f4db4075b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:10.445 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4db3fff700 1 -- 172.21.15.157:0/1789509219 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db4074fc0 con 0x7f4db4103930 2024-03-11T20:50:10.445 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4db3fff700 1 --2- 172.21.15.157:0/1789509219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4db4103930 0x7f4db4074710 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4dac0062a0 tx=0x7f4dac00cec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:50:10.445 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.446+0000 7f4db17fa700 1 -- 172.21.15.157:0/1789509219 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac021070 con 0x7f4db4103930 2024-03-11T20:50:10.445 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.447+0000 7f4db17fa700 1 -- 172.21.15.157:0/1789509219 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4dac01d8d0 con 0x7f4db4103930 2024-03-11T20:50:10.446 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.447+0000 7f4dba21b700 1 -- 172.21.15.157:0/1789509219 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db41a1650 con 0x7f4db4103930 2024-03-11T20:50:10.446 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.447+0000 7f4db17fa700 1 -- 172.21.15.157:0/1789509219 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dac0052a0 con 0x7f4db4103930 2024-03-11T20:50:10.446 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.447+0000 7f4dba21b700 1 -- 172.21.15.157:0/1789509219 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db41a1b00 con 0x7f4db4103930 2024-03-11T20:50:10.446 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.448+0000 7f4dba21b700 1 -- 172.21.15.157:0/1789509219 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db4068aa0 con 0x7f4db4103930 2024-03-11T20:50:10.446 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.448+0000 7f4db17fa700 1 -- 172.21.15.157:0/1789509219 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7f4dac024070 con 0x7f4db4103930 2024-03-11T20:50:10.447 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.449+0000 7f4db17fa700 1 --2- 172.21.15.157:0/1789509219 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4da006dd90 0x7f4da0070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:50:10.447 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.449+0000 7f4db17fa700 1 -- 172.21.15.157:0/1789509219 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4423+0+0 (secure 0 0 0) 0x7f4dac00f020 con 0x7f4db4103930 2024-03-11T20:50:10.447 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.449+0000 7f4db37fe700 1 --2- 172.21.15.157:0/1789509219 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4da006dd90 0x7f4da0070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:50:10.447 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.449+0000 7f4db37fe700 1 --2- 172.21.15.157:0/1789509219 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4da006dd90 0x7f4da0070250 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4da4005950 tx=0x7f4da400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:50:10.449 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.453+0000 7f4db17fa700 1 -- 172.21.15.157:0/1789509219 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f4dac02b020 con 0x7f4db4103930 2024-03-11T20:50:10.655 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:10.659+0000 7f4dba21b700 1 -- 172.21.15.157:0/1789509219 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f4db4062220 con 0x7f4da006dd90 2024-03-11T20:50:10.655 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:10 smithi157 ceph-mon[33359]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T20:50:10.655 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:10 smithi157 ceph-mon[33359]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:10.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:10 smithi129 ceph-mon[30184]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T20:50:10.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:10 smithi129 ceph-mon[30184]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:10.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:10 smithi129 ceph-mon[42066]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T20:50:10.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:10 smithi129 ceph-mon[42066]: pgmap v142: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 566 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:11.587 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:11 smithi157 ceph-mon[33359]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:50:11.587 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:11 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:50:11.588 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:11 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:50:11.588 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:11 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:11.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:11 smithi129 ceph-mon[30184]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:50:11.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:50:11.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:50:11.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:11 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:11.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:11 smithi129 ceph-mon[42066]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:50:11.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:11 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:50:11.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:11 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:50:11.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:11 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:12.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:12 smithi157 ceph-mon[33359]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:12.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:12 smithi129 ceph-mon[30184]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:12.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:12 smithi129 ceph-mon[42066]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:13.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.157:0/1800944777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c489abd-5f6c-4fb9-954a-aabd409fffd5"}]: dispatch 2024-03-11T20:50:13.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c489abd-5f6c-4fb9-954a-aabd409fffd5"}]: dispatch 2024-03-11T20:50:13.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c489abd-5f6c-4fb9-954a-aabd409fffd5"}]': finished 2024-03-11T20:50:13.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:13 smithi157 ceph-mon[33359]: osdmap e43: 7 total, 6 up, 7 in 2024-03-11T20:50:13.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:13 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:50:13.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.157:0/1800944777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c489abd-5f6c-4fb9-954a-aabd409fffd5"}]: dispatch 2024-03-11T20:50:13.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c489abd-5f6c-4fb9-954a-aabd409fffd5"}]: dispatch 2024-03-11T20:50:13.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c489abd-5f6c-4fb9-954a-aabd409fffd5"}]': finished 2024-03-11T20:50:13.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:13 smithi129 ceph-mon[30184]: osdmap e43: 7 total, 6 up, 7 in 2024-03-11T20:50:13.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:13 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:50:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.157:0/1800944777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c489abd-5f6c-4fb9-954a-aabd409fffd5"}]: dispatch 2024-03-11T20:50:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c489abd-5f6c-4fb9-954a-aabd409fffd5"}]: dispatch 2024-03-11T20:50:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c489abd-5f6c-4fb9-954a-aabd409fffd5"}]': finished 2024-03-11T20:50:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:13 smithi129 ceph-mon[42066]: osdmap e43: 7 total, 6 up, 7 in 2024-03-11T20:50:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:13 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:50:14.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:14 smithi129 ceph-mon[30184]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:15.036 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.157:0/2721431209' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:50:15.036 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:14 smithi129 ceph-mon[42066]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:15.036 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.157:0/2721431209' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:50:15.037 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:14 smithi157 ceph-mon[33359]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:15.037 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.157:0/2721431209' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:50:16.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:16 smithi129 ceph-mon[30184]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:16.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:16 smithi129 ceph-mon[42066]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:16.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:16 smithi157 ceph-mon[33359]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:18.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:18 smithi129 ceph-mon[30184]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:18.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:18 smithi129 ceph-mon[42066]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:18.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:18 smithi157 ceph-mon[33359]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:20.917 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:20 smithi157 ceph-mon[33359]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:20.917 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T20:50:20.917 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:20 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:20.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:20 smithi129 ceph-mon[30184]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:20.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T20:50:20.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:20 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:20.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:20 smithi129 ceph-mon[42066]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:20.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:20 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T20:50:20.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:20 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:21.944 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:21 smithi157 ceph-mon[33359]: Deploying daemon osd.6 on smithi157 2024-03-11T20:50:21.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:21 smithi129 ceph-mon[30184]: Deploying daemon osd.6 on smithi157 2024-03-11T20:50:21.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:21 smithi129 ceph-mon[42066]: Deploying daemon osd.6 on smithi157 2024-03-11T20:50:22.919 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:22 smithi157 ceph-mon[33359]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:22.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:22 smithi129 ceph-mon[30184]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:22.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:22 smithi129 ceph-mon[42066]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:24.969 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:24 smithi157 ceph-mon[33359]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:24.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:24 smithi129 ceph-mon[30184]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:24.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:24 smithi129 ceph-mon[42066]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:26.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:26 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:26.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:26 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:26.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:26 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:26.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:26 smithi157 ceph-mon[33359]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:26.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:26 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:26.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:26 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:26.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:26 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:26.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:26 smithi129 ceph-mon[30184]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:26.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:26 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:26.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:26 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:26.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:26 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:26.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:26 smithi129 ceph-mon[42066]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:28.965 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:28 smithi157 ceph-mon[33359]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:28.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:28 smithi129 ceph-mon[30184]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:28.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:28 smithi129 ceph-mon[42066]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:29.018 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:29.023+0000 7f4db17fa700 1 -- 172.21.15.157:0/1789509219 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4db4062220 con 0x7f4da006dd90 2024-03-11T20:50:29.019 INFO:teuthology.orchestra.run.smithi157.stdout:Created osd(s) 6 on host 'smithi157' 2024-03-11T20:50:29.022 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:29.026+0000 7f4dba21b700 1 -- 172.21.15.157:0/1789509219 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4da006dd90 msgr2=0x7f4da0070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:29.022 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:29.026+0000 7f4dba21b700 1 --2- 172.21.15.157:0/1789509219 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4da006dd90 0x7f4da0070250 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4da4005950 tx=0x7f4da400a300 comp rx=0 tx=0).stop 2024-03-11T20:50:29.022 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:29.026+0000 7f4dba21b700 1 -- 172.21.15.157:0/1789509219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4db4103930 msgr2=0x7f4db4074710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:29.022 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:29.026+0000 7f4dba21b700 1 --2- 172.21.15.157:0/1789509219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4db4103930 0x7f4db4074710 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4dac0062a0 tx=0x7f4dac00cec0 comp rx=0 tx=0).stop 2024-03-11T20:50:29.023 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:29.026+0000 7f4dba21b700 1 -- 172.21.15.157:0/1789509219 shutdown_connections 2024-03-11T20:50:29.023 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:29.026+0000 7f4dba21b700 1 --2- 172.21.15.157:0/1789509219 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4da006dd90 0x7f4da0070250 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:29.023 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:29.026+0000 7f4dba21b700 1 --2- 172.21.15.157:0/1789509219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4db4103930 0x7f4db4074710 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:29.023 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:29.026+0000 7f4dba21b700 1 --2- 172.21.15.157:0/1789509219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4db4101e10 0x7f4db4075b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:29.023 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:29.026+0000 7f4dba21b700 1 --2- 172.21.15.157:0/1789509219 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4db4102f70 0x7f4db40741d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:29.023 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:29.026+0000 7f4dba21b700 1 -- 172.21.15.157:0/1789509219 >> 172.21.15.157:0/1789509219 conn(0x7f4db40fd3f0 msgr2=0x7f4db40feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:50:29.023 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:29.027+0000 7f4dba21b700 1 -- 172.21.15.157:0/1789509219 shutdown_connections 2024-03-11T20:50:29.023 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:29.027+0000 7f4dba21b700 1 -- 172.21.15.157:0/1789509219 wait complete. 2024-03-11T20:50:29.604 DEBUG:teuthology.orchestra.run.smithi157:osd.6> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.6.service 2024-03-11T20:50:29.606 INFO:tasks.cephadm:Deploying osd.7 on smithi157 with /dev/vg_nvme/lv_1... 2024-03-11T20:50:29.607 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T20:50:29.894 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:50:29.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:29 smithi129 ceph-mon[30184]: from='osd.6 [v2:172.21.15.157:6816/2710731257,v1:172.21.15.157:6817/2710731257]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T20:50:29.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:29 smithi129 ceph-mon[30184]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T20:50:29.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:29.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:29 smithi129 ceph-mon[42066]: from='osd.6 [v2:172.21.15.157:6816/2710731257,v1:172.21.15.157:6817/2710731257]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T20:50:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:29 smithi129 ceph-mon[42066]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T20:50:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:29.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:29 smithi157 ceph-mon[33359]: from='osd.6 [v2:172.21.15.157:6816/2710731257,v1:172.21.15.157:6817/2710731257]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T20:50:29.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:29 smithi157 ceph-mon[33359]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T20:50:29.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:29.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:30.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:30 smithi129 ceph-mon[30184]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:30.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:30 smithi129 ceph-mon[30184]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T20:50:30.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:30 smithi129 ceph-mon[30184]: from='osd.6 [v2:172.21.15.157:6816/2710731257,v1:172.21.15.157:6817/2710731257]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:30.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:30 smithi129 ceph-mon[30184]: osdmap e44: 7 total, 6 up, 7 in 2024-03-11T20:50:30.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:30 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:50:30.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:30 smithi129 ceph-mon[30184]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:30.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:30 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:30.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:30 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:30.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:30 smithi129 ceph-mon[42066]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:30.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:30 smithi129 ceph-mon[42066]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T20:50:30.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:30 smithi129 ceph-mon[42066]: from='osd.6 [v2:172.21.15.157:6816/2710731257,v1:172.21.15.157:6817/2710731257]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:30.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:30 smithi129 ceph-mon[42066]: osdmap e44: 7 total, 6 up, 7 in 2024-03-11T20:50:30.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:30 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:50:30.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:30 smithi129 ceph-mon[42066]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:30.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:30 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:30.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:30 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:30.986 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 20:50:30 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T20:50:30.681+0000 7fe643c26700 -1 osd.6 0 waiting for initial osdmap 2024-03-11T20:50:30.986 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 20:50:30 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T20:50:30.688+0000 7fe63fa19700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T20:50:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:30 smithi157 ceph-mon[33359]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:30 smithi157 ceph-mon[33359]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T20:50:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:30 smithi157 ceph-mon[33359]: from='osd.6 [v2:172.21.15.157:6816/2710731257,v1:172.21.15.157:6817/2710731257]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:30 smithi157 ceph-mon[33359]: osdmap e44: 7 total, 6 up, 7 in 2024-03-11T20:50:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:30 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:50:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:30 smithi157 ceph-mon[33359]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:30 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:30 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:31.909 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T20:50:31.909 INFO:teuthology.orchestra.run.smithi157.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T20:50:31.909 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10+0 records in 2024-03-11T20:50:31.909 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records out 2024-03-11T20:50:31.909 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00983036 s, 1.1 GB/s 2024-03-11T20:50:31.909 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping successful for: 2024-03-11T20:50:31.934 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:31 smithi157 ceph-mon[33359]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-03-11T20:50:31.934 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:31 smithi157 ceph-mon[33359]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T20:50:31.934 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:31 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:50:31.934 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:31 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:50:31.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:31 smithi129 ceph-mon[30184]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-03-11T20:50:31.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:31 smithi129 ceph-mon[30184]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T20:50:31.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:31 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:50:31.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:31 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:50:31.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:31 smithi129 ceph-mon[42066]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-03-11T20:50:31.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:31 smithi129 ceph-mon[42066]: osdmap e45: 7 total, 6 up, 7 in 2024-03-11T20:50:31.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:31 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:50:31.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:31 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:50:32.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:32 smithi129 ceph-mon[30184]: purged_snaps scrub starts 2024-03-11T20:50:32.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:32 smithi129 ceph-mon[30184]: purged_snaps scrub ok 2024-03-11T20:50:32.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:32 smithi129 ceph-mon[30184]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:32.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:32 smithi129 ceph-mon[30184]: osd.6 [v2:172.21.15.157:6816/2710731257,v1:172.21.15.157:6817/2710731257] boot 2024-03-11T20:50:32.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:32 smithi129 ceph-mon[30184]: osdmap e46: 7 total, 7 up, 7 in 2024-03-11T20:50:32.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:32 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:50:32.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:32 smithi129 ceph-mon[42066]: purged_snaps scrub starts 2024-03-11T20:50:32.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:32 smithi129 ceph-mon[42066]: purged_snaps scrub ok 2024-03-11T20:50:32.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:32 smithi129 ceph-mon[42066]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:32.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:32 smithi129 ceph-mon[42066]: osd.6 [v2:172.21.15.157:6816/2710731257,v1:172.21.15.157:6817/2710731257] boot 2024-03-11T20:50:32.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:32 smithi129 ceph-mon[42066]: osdmap e46: 7 total, 7 up, 7 in 2024-03-11T20:50:32.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:32 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:50:32.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:32 smithi157 ceph-mon[33359]: purged_snaps scrub starts 2024-03-11T20:50:32.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:32 smithi157 ceph-mon[33359]: purged_snaps scrub ok 2024-03-11T20:50:32.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:32 smithi157 ceph-mon[33359]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-11T20:50:32.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:32 smithi157 ceph-mon[33359]: osd.6 [v2:172.21.15.157:6816/2710731257,v1:172.21.15.157:6817/2710731257] boot 2024-03-11T20:50:32.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:32 smithi157 ceph-mon[33359]: osdmap e46: 7 total, 7 up, 7 in 2024-03-11T20:50:32.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:32 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:50:33.157 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi157:vg_nvme/lv_1 2024-03-11T20:50:33.740 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:50:33.968 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:33 smithi157 ceph-mon[33359]: osdmap e47: 7 total, 7 up, 7 in 2024-03-11T20:50:33.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:33 smithi129 ceph-mon[30184]: osdmap e47: 7 total, 7 up, 7 in 2024-03-11T20:50:33.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:33 smithi129 ceph-mon[42066]: osdmap e47: 7 total, 7 up, 7 in 2024-03-11T20:50:34.838 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:34 smithi157 ceph-mon[33359]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:34.839 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:34 smithi157 ceph-mon[33359]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T20:50:34.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:34 smithi129 ceph-mon[30184]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:34.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:34 smithi129 ceph-mon[30184]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T20:50:34.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:34 smithi129 ceph-mon[42066]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:34.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:34 smithi129 ceph-mon[42066]: osdmap e48: 7 total, 7 up, 7 in 2024-03-11T20:50:35.862 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:35 smithi157 ceph-mon[33359]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T20:50:35.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:35 smithi129 ceph-mon[30184]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T20:50:35.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:35 smithi129 ceph-mon[42066]: osdmap e49: 7 total, 7 up, 7 in 2024-03-11T20:50:36.390 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.390+0000 7f1e23aa9700 1 -- 172.21.15.157:0/3913300711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e1c107af0 msgr2=0x7f1e1c109f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:36.390 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e23aa9700 1 --2- 172.21.15.157:0/3913300711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e1c107af0 0x7f1e1c109f60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1e1400b0a0 tx=0x7f1e1400b3b0 comp rx=0 tx=0).stop 2024-03-11T20:50:36.390 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e23aa9700 1 -- 172.21.15.157:0/3913300711 shutdown_connections 2024-03-11T20:50:36.391 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e23aa9700 1 --2- 172.21.15.157:0/3913300711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e1c107af0 0x7f1e1c109f60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:36.391 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e23aa9700 1 --2- 172.21.15.157:0/3913300711 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1e1c102890 0x7f1e1c104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:36.391 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e23aa9700 1 --2- 172.21.15.157:0/3913300711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e1c1051c0 0x7f1e1c1075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:36.391 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e23aa9700 1 -- 172.21.15.157:0/3913300711 >> 172.21.15.157:0/3913300711 conn(0x7f1e1c0fc2e0 msgr2=0x7f1e1c0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:50:36.391 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e23aa9700 1 -- 172.21.15.157:0/3913300711 shutdown_connections 2024-03-11T20:50:36.391 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e23aa9700 1 -- 172.21.15.157:0/3913300711 wait complete. 2024-03-11T20:50:36.391 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e23aa9700 1 Processor -- start 2024-03-11T20:50:36.391 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e23aa9700 1 -- start start 2024-03-11T20:50:36.391 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e23aa9700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1e1c102890 0x7f1e1c197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:50:36.391 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e23aa9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e1c1051c0 0x7f1e1c1979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:50:36.392 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e23aa9700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e1c107af0 0x7f1e1c19ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:50:36.392 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e23aa9700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c10ce50 con 0x7f1e1c107af0 2024-03-11T20:50:36.392 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e23aa9700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c10ccd0 con 0x7f1e1c1051c0 2024-03-11T20:50:36.392 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e23aa9700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c10cfd0 con 0x7f1e1c102890 2024-03-11T20:50:36.392 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.391+0000 7f1e21044700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e1c1051c0 0x7f1e1c1979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:50:36.392 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.392+0000 7f1e21044700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e1c1051c0 0x7f1e1c1979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:51522/0 (socket says 172.21.15.157:51522) 2024-03-11T20:50:36.392 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.392+0000 7f1e21044700 1 -- 172.21.15.157:0/3299318782 learned_addr learned my addr 172.21.15.157:0/3299318782 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:50:36.392 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.392+0000 7f1e21845700 1 --2- 172.21.15.157:0/3299318782 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1e1c102890 0x7f1e1c197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:50:36.392 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.392+0000 7f1e22046700 1 --2- 172.21.15.157:0/3299318782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e1c107af0 0x7f1e1c19ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:50:36.392 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.392+0000 7f1e21044700 1 -- 172.21.15.157:0/3299318782 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1e1c102890 msgr2=0x7f1e1c197480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:36.392 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.392+0000 7f1e21044700 1 --2- 172.21.15.157:0/3299318782 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1e1c102890 0x7f1e1c197480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:36.392 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.392+0000 7f1e21044700 1 -- 172.21.15.157:0/3299318782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e1c107af0 msgr2=0x7f1e1c19ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:36.393 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.392+0000 7f1e21044700 1 --2- 172.21.15.157:0/3299318782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e1c107af0 0x7f1e1c19ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:36.393 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.392+0000 7f1e21044700 1 -- 172.21.15.157:0/3299318782 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c19f0e0 con 0x7f1e1c1051c0 2024-03-11T20:50:36.393 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.392+0000 7f1e21044700 1 --2- 172.21.15.157:0/3299318782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e1c1051c0 0x7f1e1c1979c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00c9b0 tx=0x7f1e0c00ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:50:36.393 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.392+0000 7f1e12ffd700 1 -- 172.21.15.157:0/3299318782 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c007b00 con 0x7f1e1c1051c0 2024-03-11T20:50:36.393 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.392+0000 7f1e23aa9700 1 -- 172.21.15.157:0/3299318782 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e1c19f3d0 con 0x7f1e1c1051c0 2024-03-11T20:50:36.393 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.392+0000 7f1e12ffd700 1 -- 172.21.15.157:0/3299318782 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f1e0c00ce50 con 0x7f1e1c1051c0 2024-03-11T20:50:36.393 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.392+0000 7f1e12ffd700 1 -- 172.21.15.157:0/3299318782 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c018400 con 0x7f1e1c1051c0 2024-03-11T20:50:36.393 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.392+0000 7f1e23aa9700 1 -- 172.21.15.157:0/3299318782 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e1c19f960 con 0x7f1e1c1051c0 2024-03-11T20:50:36.393 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.393+0000 7f1e23aa9700 1 -- 172.21.15.157:0/3299318782 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e1c068aa0 con 0x7f1e1c1051c0 2024-03-11T20:50:36.394 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.394+0000 7f1e12ffd700 1 -- 172.21.15.157:0/3299318782 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7f1e0c00f450 con 0x7f1e1c1051c0 2024-03-11T20:50:36.394 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.394+0000 7f1e12ffd700 1 --2- 172.21.15.157:0/3299318782 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f1e0806dd90 0x7f1e08070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:50:36.394 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.394+0000 7f1e12ffd700 1 -- 172.21.15.157:0/3299318782 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4855+0+0 (secure 0 0 0) 0x7f1e0c08d580 con 0x7f1e1c1051c0 2024-03-11T20:50:36.394 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.394+0000 7f1e21845700 1 --2- 172.21.15.157:0/3299318782 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f1e0806dd90 0x7f1e08070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:50:36.394 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.395+0000 7f1e21845700 1 --2- 172.21.15.157:0/3299318782 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f1e0806dd90 0x7f1e08070250 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1e18004630 tx=0x7f1e1800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:50:36.396 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.399+0000 7f1e12ffd700 1 -- 172.21.15.157:0/3299318782 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f1e0c014030 con 0x7f1e1c1051c0 2024-03-11T20:50:36.598 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:36.601+0000 7f1e23aa9700 1 -- 172.21.15.157:0/3299318782 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e1c062220 con 0x7f1e0806dd90 2024-03-11T20:50:36.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: pgmap v162: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:36.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: osdmap e50: 7 total, 7 up, 7 in 2024-03-11T20:50:36.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:36.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:36.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:36.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:36.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:36.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:36.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:36.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:36.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:36.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:36.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:36.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:36.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:36.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:50:36.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:50:36.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: pgmap v162: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: osdmap e50: 7 total, 7 up, 7 in 2024-03-11T20:50:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:36.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:36.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:36.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:36.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:36.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:36.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:36.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:36.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:36.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:50:36.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:50:36.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:36 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:36.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: pgmap v162: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:36.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: osdmap e50: 7 total, 7 up, 7 in 2024-03-11T20:50:36.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:36.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:36.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T20:50:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T20:50:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:36 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:37.887 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:37 smithi157 ceph-mon[33359]: Detected new or changed devices on smithi157 2024-03-11T20:50:37.888 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:37 smithi157 ceph-mon[33359]: Adjusting osd_memory_target on smithi157 to 5696M 2024-03-11T20:50:37.888 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:37 smithi157 ceph-mon[33359]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:50:37.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:37 smithi129 ceph-mon[42066]: Detected new or changed devices on smithi157 2024-03-11T20:50:37.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:37 smithi129 ceph-mon[42066]: Adjusting osd_memory_target on smithi157 to 5696M 2024-03-11T20:50:37.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:37 smithi129 ceph-mon[42066]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:50:37.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:37 smithi129 ceph-mon[30184]: Detected new or changed devices on smithi157 2024-03-11T20:50:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:37 smithi129 ceph-mon[30184]: Adjusting osd_memory_target on smithi157 to 5696M 2024-03-11T20:50:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:37 smithi129 ceph-mon[30184]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:50:38.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:38 smithi129 ceph-mon[30184]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:38.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:38 smithi129 ceph-mon[42066]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:38.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:38 smithi157 ceph-mon[33359]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.157:0/61986962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22bacf1f-35c5-4dee-81a6-92dff2f076c2"}]: dispatch 2024-03-11T20:50:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22bacf1f-35c5-4dee-81a6-92dff2f076c2"}]: dispatch 2024-03-11T20:50:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22bacf1f-35c5-4dee-81a6-92dff2f076c2"}]': finished 2024-03-11T20:50:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:39 smithi129 ceph-mon[30184]: osdmap e51: 8 total, 7 up, 8 in 2024-03-11T20:50:39.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:39 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:50:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.157:0/61986962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22bacf1f-35c5-4dee-81a6-92dff2f076c2"}]: dispatch 2024-03-11T20:50:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22bacf1f-35c5-4dee-81a6-92dff2f076c2"}]: dispatch 2024-03-11T20:50:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22bacf1f-35c5-4dee-81a6-92dff2f076c2"}]': finished 2024-03-11T20:50:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:39 smithi129 ceph-mon[42066]: osdmap e51: 8 total, 7 up, 8 in 2024-03-11T20:50:39.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:39 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:50:39.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.157:0/61986962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22bacf1f-35c5-4dee-81a6-92dff2f076c2"}]: dispatch 2024-03-11T20:50:39.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22bacf1f-35c5-4dee-81a6-92dff2f076c2"}]: dispatch 2024-03-11T20:50:39.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22bacf1f-35c5-4dee-81a6-92dff2f076c2"}]': finished 2024-03-11T20:50:39.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:39 smithi157 ceph-mon[33359]: osdmap e51: 8 total, 7 up, 8 in 2024-03-11T20:50:39.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:39 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:50:40.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:40 smithi129 ceph-mon[42066]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T20:50:40.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.157:0/2467018648' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:50:40.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:40 smithi129 ceph-mon[30184]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T20:50:40.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.157:0/2467018648' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:50:40.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:40 smithi157 ceph-mon[33359]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T20:50:40.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.157:0/2467018648' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T20:50:42.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:42 smithi129 ceph-mon[30184]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-11T20:50:42.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:42 smithi129 ceph-mon[42066]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-11T20:50:42.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:42 smithi157 ceph-mon[33359]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-11T20:50:44.894 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:44 smithi157 ceph-mon[33359]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T20:50:44.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:44 smithi129 ceph-mon[30184]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T20:50:44.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:44 smithi129 ceph-mon[42066]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T20:50:46.874 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:46 smithi157 ceph-mon[33359]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:50:46.875 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:46 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T20:50:46.875 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:46 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:46.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:46 smithi129 ceph-mon[30184]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:50:46.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T20:50:46.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:46 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:46.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:46 smithi129 ceph-mon[42066]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:50:46.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:46 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T20:50:46.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:46 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:47.822 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:47 smithi157 ceph-mon[33359]: Deploying daemon osd.7 on smithi157 2024-03-11T20:50:48.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:47 smithi129 ceph-mon[30184]: Deploying daemon osd.7 on smithi157 2024-03-11T20:50:48.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:47 smithi129 ceph-mon[42066]: Deploying daemon osd.7 on smithi157 2024-03-11T20:50:48.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:48 smithi157 ceph-mon[33359]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:50:49.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:48 smithi129 ceph-mon[30184]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:50:49.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:48 smithi129 ceph-mon[42066]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-11T20:50:50.860 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:50 smithi157 ceph-mon[33359]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:51.030 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:50 smithi129 ceph-mon[30184]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:51.031 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:50 smithi129 ceph-mon[42066]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:52.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:52 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:52.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:52 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:52.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:52 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:52.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:52 smithi129 ceph-mon[30184]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:52.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:52 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:52.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:52 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:52.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:52 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:52.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:52 smithi129 ceph-mon[42066]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:52.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:52 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:52.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:52 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:52.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:52 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:52.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:52 smithi157 ceph-mon[33359]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:54.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:54 smithi129 ceph-mon[30184]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:54.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:54.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:54 smithi129 ceph-mon[30184]: from='osd.7 [v2:172.21.15.157:6824/1786686696,v1:172.21.15.157:6825/1786686696]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T20:50:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:54 smithi129 ceph-mon[30184]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T20:50:54.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:54 smithi129 ceph-mon[42066]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:54.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:54.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:54.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:54.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:54.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:54.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:54 smithi129 ceph-mon[42066]: from='osd.7 [v2:172.21.15.157:6824/1786686696,v1:172.21.15.157:6825/1786686696]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T20:50:54.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:54 smithi129 ceph-mon[42066]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T20:50:54.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:54 smithi157 ceph-mon[33359]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:54.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:54.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:54.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:54.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:54.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:54.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:54 smithi157 ceph-mon[33359]: from='osd.7 [v2:172.21.15.157:6824/1786686696,v1:172.21.15.157:6825/1786686696]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T20:50:54.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:54 smithi157 ceph-mon[33359]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T20:50:55.413 INFO:teuthology.orchestra.run.smithi157.stdout:Created osd(s) 7 on host 'smithi157' 2024-03-11T20:50:55.413 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:55.417+0000 7f1e12ffd700 1 -- 172.21.15.157:0/3299318782 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1e1c062220 con 0x7f1e0806dd90 2024-03-11T20:50:55.415 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:55.419+0000 7f1e23aa9700 1 -- 172.21.15.157:0/3299318782 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f1e0806dd90 msgr2=0x7f1e08070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:55.415 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:55.419+0000 7f1e23aa9700 1 --2- 172.21.15.157:0/3299318782 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f1e0806dd90 0x7f1e08070250 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1e18004630 tx=0x7f1e1800b3c0 comp rx=0 tx=0).stop 2024-03-11T20:50:55.415 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:55.419+0000 7f1e23aa9700 1 -- 172.21.15.157:0/3299318782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e1c1051c0 msgr2=0x7f1e1c1979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:55.415 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:55.419+0000 7f1e23aa9700 1 --2- 172.21.15.157:0/3299318782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e1c1051c0 0x7f1e1c1979c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00c9b0 tx=0x7f1e0c00ccc0 comp rx=0 tx=0).stop 2024-03-11T20:50:55.416 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:55.419+0000 7f1e23aa9700 1 -- 172.21.15.157:0/3299318782 shutdown_connections 2024-03-11T20:50:55.416 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:55.419+0000 7f1e23aa9700 1 --2- 172.21.15.157:0/3299318782 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f1e0806dd90 0x7f1e08070250 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:55.416 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:55.419+0000 7f1e23aa9700 1 --2- 172.21.15.157:0/3299318782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e1c107af0 0x7f1e1c19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:55.416 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:55.419+0000 7f1e23aa9700 1 --2- 172.21.15.157:0/3299318782 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1e1c102890 0x7f1e1c197480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:55.416 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:55.419+0000 7f1e23aa9700 1 --2- 172.21.15.157:0/3299318782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e1c1051c0 0x7f1e1c1979c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:55.416 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:55.419+0000 7f1e23aa9700 1 -- 172.21.15.157:0/3299318782 >> 172.21.15.157:0/3299318782 conn(0x7f1e1c0fc2e0 msgr2=0x7f1e1c103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:50:55.416 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:55.419+0000 7f1e23aa9700 1 -- 172.21.15.157:0/3299318782 shutdown_connections 2024-03-11T20:50:55.416 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:50:55.419+0000 7f1e23aa9700 1 -- 172.21.15.157:0/3299318782 wait complete. 2024-03-11T20:50:56.012 DEBUG:teuthology.orchestra.run.smithi157:osd.7> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.7.service 2024-03-11T20:50:56.014 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-11T20:50:56.015 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T20:50:56.265 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:50:56.655 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:56.655 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:56.655 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:56.655 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[30184]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T20:50:56.655 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[30184]: from='osd.7 [v2:172.21.15.157:6824/1786686696,v1:172.21.15.157:6825/1786686696]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:56.655 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[30184]: osdmap e52: 8 total, 7 up, 8 in 2024-03-11T20:50:56.655 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:50:56.655 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[30184]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:56.655 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[30184]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:56.656 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:56.656 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:56.656 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:56.656 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[42066]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T20:50:56.656 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[42066]: from='osd.7 [v2:172.21.15.157:6824/1786686696,v1:172.21.15.157:6825/1786686696]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:56.656 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[42066]: osdmap e52: 8 total, 7 up, 8 in 2024-03-11T20:50:56.656 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:50:56.656 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[42066]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:56.656 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:56 smithi129 ceph-mon[42066]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:56.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:56 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:56.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:56 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:56.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:56 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:50:56.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:56 smithi157 ceph-mon[33359]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T20:50:56.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:56 smithi157 ceph-mon[33359]: from='osd.7 [v2:172.21.15.157:6824/1786686696,v1:172.21.15.157:6825/1786686696]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:56.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:56 smithi157 ceph-mon[33359]: osdmap e52: 8 total, 7 up, 8 in 2024-03-11T20:50:56.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:56 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:50:56.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:56 smithi157 ceph-mon[33359]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2024-03-11T20:50:56.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:56 smithi157 ceph-mon[33359]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-11T20:50:56.736 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 20:50:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T20:50:56.454+0000 7f25936c9700 -1 osd.7 0 waiting for initial osdmap 2024-03-11T20:50:56.736 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 20:50:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T20:50:56.461+0000 7f258f4bc700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T20:50:57.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:57 smithi129 ceph-mon[42066]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-03-11T20:50:57.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:57 smithi129 ceph-mon[42066]: osdmap e53: 8 total, 7 up, 8 in 2024-03-11T20:50:57.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:57 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:50:57.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:57 smithi129 ceph-mon[30184]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-03-11T20:50:57.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:57 smithi129 ceph-mon[30184]: osdmap e53: 8 total, 7 up, 8 in 2024-03-11T20:50:57.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:57 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:50:57.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:57 smithi157 ceph-mon[33359]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2024-03-11T20:50:57.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:57 smithi157 ceph-mon[33359]: osdmap e53: 8 total, 7 up, 8 in 2024-03-11T20:50:57.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:57 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:50:58.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.037+0000 7f8765d18700 1 -- 172.21.15.129:0/2865152410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8760101f10 msgr2=0x7f8760102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:58.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.037+0000 7f8765d18700 1 --2- 172.21.15.129:0/2865152410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8760101f10 0x7f8760102370 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f87540099c0 tx=0x7f8754009cd0 comp rx=0 tx=0).stop 2024-03-11T20:50:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.037+0000 7f8765d18700 1 -- 172.21.15.129:0/2865152410 shutdown_connections 2024-03-11T20:50:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.037+0000 7f8765d18700 1 --2- 172.21.15.129:0/2865152410 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f87601028b0 0x7f8760109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.037+0000 7f8765d18700 1 --2- 172.21.15.129:0/2865152410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8760100d10 0x7f8760101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.037+0000 7f8765d18700 1 --2- 172.21.15.129:0/2865152410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8760101f10 0x7f8760102370 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.037+0000 7f8765d18700 1 -- 172.21.15.129:0/2865152410 >> 172.21.15.129:0/2865152410 conn(0x7f87600fc2d0 msgr2=0x7f87600fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:50:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.037+0000 7f8765d18700 1 -- 172.21.15.129:0/2865152410 shutdown_connections 2024-03-11T20:50:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.037+0000 7f8765d18700 1 -- 172.21.15.129:0/2865152410 wait complete. 2024-03-11T20:50:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.037+0000 7f8765d18700 1 Processor -- start 2024-03-11T20:50:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.037+0000 7f8765d18700 1 -- start start 2024-03-11T20:50:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f8765d18700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8760100d10 0x7f8760199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:50:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f8765d18700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8760101f10 0x7f8760199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:50:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f8765d18700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87601028b0 0x7f87601a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:50:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f8765d18700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876010bd70 con 0x7f87601028b0 2024-03-11T20:50:58.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f8765d18700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876010bbf0 con 0x7f8760101f10 2024-03-11T20:50:58.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f8765d18700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876010bef0 con 0x7f8760100d10 2024-03-11T20:50:58.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f875f7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8760100d10 0x7f8760199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:50:58.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f875ffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87601028b0 0x7f87601a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:50:58.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f875f7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8760100d10 0x7f8760199720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:56788/0 (socket says 172.21.15.129:56788) 2024-03-11T20:50:58.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f875f7fe700 1 -- 172.21.15.129:0/235041790 learned_addr learned my addr 172.21.15.129:0/235041790 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:50:58.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f875effd700 1 --2- 172.21.15.129:0/235041790 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8760101f10 0x7f8760199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:50:58.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f875ffff700 1 -- 172.21.15.129:0/235041790 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8760100d10 msgr2=0x7f8760199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:58.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f875ffff700 1 --2- 172.21.15.129:0/235041790 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8760100d10 0x7f8760199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:58.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f875ffff700 1 -- 172.21.15.129:0/235041790 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8760101f10 msgr2=0x7f8760199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:58.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f875ffff700 1 --2- 172.21.15.129:0/235041790 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8760101f10 0x7f8760199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:58.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f875ffff700 1 -- 172.21.15.129:0/235041790 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87601a1380 con 0x7f87601028b0 2024-03-11T20:50:58.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f875f7fe700 1 --2- 172.21.15.129:0/235041790 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8760100d10 0x7f8760199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:50:58.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f875ffff700 1 --2- 172.21.15.129:0/235041790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87601028b0 0x7f87601a0ce0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8758009f60 tx=0x7f8758009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:50:58.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f875cff9700 1 -- 172.21.15.129:0/235041790 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87580092e0 con 0x7f87601028b0 2024-03-11T20:50:58.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f875cff9700 1 -- 172.21.15.129:0/235041790 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8758004bb0 con 0x7f87601028b0 2024-03-11T20:50:58.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f8765d18700 1 -- 172.21.15.129:0/235041790 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87601a1610 con 0x7f87601028b0 2024-03-11T20:50:58.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f875cff9700 1 -- 172.21.15.129:0/235041790 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758004020 con 0x7f87601028b0 2024-03-11T20:50:58.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.038+0000 7f8765d18700 1 -- 172.21.15.129:0/235041790 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87601a1b00 con 0x7f87601028b0 2024-03-11T20:50:58.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.039+0000 7f8765d18700 1 -- 172.21.15.129:0/235041790 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8760068aa0 con 0x7f87601028b0 2024-03-11T20:50:58.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.040+0000 7f875cff9700 1 -- 172.21.15.129:0/235041790 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7f875801b030 con 0x7f87601028b0 2024-03-11T20:50:58.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.040+0000 7f875cff9700 1 --2- 172.21.15.129:0/235041790 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f874806dd90 0x7f8748070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:50:58.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.040+0000 7f875cff9700 1 -- 172.21.15.129:0/235041790 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5320+0+0 (secure 0 0 0) 0x7f875808d360 con 0x7f87601028b0 2024-03-11T20:50:58.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.040+0000 7f875f7fe700 1 --2- 172.21.15.129:0/235041790 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f874806dd90 0x7f8748070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:50:58.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.041+0000 7f875f7fe700 1 --2- 172.21.15.129:0/235041790 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f874806dd90 0x7f8748070250 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f87500105c0 tx=0x7f875000b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:50:58.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.045+0000 7f875cff9700 1 -- 172.21.15.129:0/235041790 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f8758022020 con 0x7f87601028b0 2024-03-11T20:50:58.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.240+0000 7f8765d18700 1 -- 172.21.15.129:0/235041790 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f876002d830 con 0x7f87601028b0 2024-03-11T20:50:58.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.240+0000 7f875cff9700 1 -- 172.21.15.129:0/235041790 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v54) v1 ==== 74+0+130 (secure 0 0 0) 0x7f8758010040 con 0x7f87601028b0 2024-03-11T20:50:58.236 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:50:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.242+0000 7f8765d18700 1 -- 172.21.15.129:0/235041790 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f874806dd90 msgr2=0x7f8748070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.242+0000 7f8765d18700 1 --2- 172.21.15.129:0/235041790 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f874806dd90 0x7f8748070250 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f87500105c0 tx=0x7f875000b4a0 comp rx=0 tx=0).stop 2024-03-11T20:50:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.242+0000 7f8765d18700 1 -- 172.21.15.129:0/235041790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87601028b0 msgr2=0x7f87601a0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:50:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.242+0000 7f8765d18700 1 --2- 172.21.15.129:0/235041790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87601028b0 0x7f87601a0ce0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8758009f60 tx=0x7f8758009f90 comp rx=0 tx=0).stop 2024-03-11T20:50:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.242+0000 7f8765d18700 1 -- 172.21.15.129:0/235041790 shutdown_connections 2024-03-11T20:50:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.242+0000 7f8765d18700 1 --2- 172.21.15.129:0/235041790 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f874806dd90 0x7f8748070250 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.242+0000 7f8765d18700 1 --2- 172.21.15.129:0/235041790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87601028b0 0x7f87601a0ce0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.242+0000 7f8765d18700 1 --2- 172.21.15.129:0/235041790 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8760100d10 0x7f8760199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:58.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.242+0000 7f8765d18700 1 --2- 172.21.15.129:0/235041790 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8760101f10 0x7f8760199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:50:58.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.242+0000 7f8765d18700 1 -- 172.21.15.129:0/235041790 >> 172.21.15.129:0/235041790 conn(0x7f87600fc2d0 msgr2=0x7f876006ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:50:58.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.242+0000 7f8765d18700 1 -- 172.21.15.129:0/235041790 shutdown_connections 2024-03-11T20:50:58.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:50:58.242+0000 7f8765d18700 1 -- 172.21.15.129:0/235041790 wait complete. 2024-03-11T20:50:58.547 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:58 smithi157 ceph-mon[33359]: purged_snaps scrub starts 2024-03-11T20:50:58.548 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:58 smithi157 ceph-mon[33359]: purged_snaps scrub ok 2024-03-11T20:50:58.548 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:58 smithi157 ceph-mon[33359]: osd.7 [v2:172.21.15.157:6824/1786686696,v1:172.21.15.157:6825/1786686696] boot 2024-03-11T20:50:58.548 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:58 smithi157 ceph-mon[33359]: osdmap e54: 8 total, 8 up, 8 in 2024-03-11T20:50:58.548 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:58 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:50:58.548 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:58 smithi157 ceph-mon[33359]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:58.548 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/235041790' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T20:50:58.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:58 smithi129 ceph-mon[42066]: purged_snaps scrub starts 2024-03-11T20:50:58.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:58 smithi129 ceph-mon[42066]: purged_snaps scrub ok 2024-03-11T20:50:58.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:58 smithi129 ceph-mon[42066]: osd.7 [v2:172.21.15.157:6824/1786686696,v1:172.21.15.157:6825/1786686696] boot 2024-03-11T20:50:58.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:58 smithi129 ceph-mon[42066]: osdmap e54: 8 total, 8 up, 8 in 2024-03-11T20:50:58.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:58 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:50:58.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:58 smithi129 ceph-mon[42066]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:58.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/235041790' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T20:50:58.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:58 smithi129 ceph-mon[30184]: purged_snaps scrub starts 2024-03-11T20:50:58.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:58 smithi129 ceph-mon[30184]: purged_snaps scrub ok 2024-03-11T20:50:58.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:58 smithi129 ceph-mon[30184]: osd.7 [v2:172.21.15.157:6824/1786686696,v1:172.21.15.157:6825/1786686696] boot 2024-03-11T20:50:58.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:58 smithi129 ceph-mon[30184]: osdmap e54: 8 total, 8 up, 8 in 2024-03-11T20:50:58.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:58 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:50:58.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:58 smithi129 ceph-mon[30184]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:50:58.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/235041790' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T20:50:58.908 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1710190257,"num_in_osds":8,"osd_in_since":1710190239,"num_remapped_pgs":1} 2024-03-11T20:50:58.908 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T20:50:59.151 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:50:59.517 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[30184]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T20:50:59.517 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:59.517 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:59.517 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:59.517 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:59.517 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:59.517 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:59.517 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:59.517 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:59.517 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:59.518 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:59.518 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[42066]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T20:50:59.518 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:59.518 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:59.518 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:59.518 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:59.518 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:59.518 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:59.518 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:59.518 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:59.518 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:59.519 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:50:59 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:59.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:59 smithi157 ceph-mon[33359]: osdmap e55: 8 total, 8 up, 8 in 2024-03-11T20:50:59.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:59 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:59.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:59 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:59.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:59 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:59.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:59 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:59.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:59 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:59.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:59 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:50:59.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:59 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:50:59.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:59 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:50:59.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:59 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:50:59.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:50:59 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:00.515 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:00 smithi129 ceph-mon[30184]: Detected new or changed devices on smithi157 2024-03-11T20:51:00.515 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:00 smithi129 ceph-mon[30184]: Adjusting osd_memory_target on smithi157 to 4272M 2024-03-11T20:51:00.515 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:00 smithi129 ceph-mon[30184]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T20:51:00.515 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:00 smithi129 ceph-mon[30184]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:51:00.515 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:00 smithi129 ceph-mon[42066]: Detected new or changed devices on smithi157 2024-03-11T20:51:00.515 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:00 smithi129 ceph-mon[42066]: Adjusting osd_memory_target on smithi157 to 4272M 2024-03-11T20:51:00.515 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:00 smithi129 ceph-mon[42066]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T20:51:00.515 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:00 smithi129 ceph-mon[42066]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:51:00.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:00 smithi157 ceph-mon[33359]: Detected new or changed devices on smithi157 2024-03-11T20:51:00.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:00 smithi157 ceph-mon[33359]: Adjusting osd_memory_target on smithi157 to 4272M 2024-03-11T20:51:00.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:00 smithi157 ceph-mon[33359]: osdmap e56: 8 total, 8 up, 8 in 2024-03-11T20:51:00.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:00 smithi157 ceph-mon[33359]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:51:00.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.747+0000 7ff6a94f1700 1 -- 172.21.15.129:0/1434993871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6a4101680 msgr2=0x7ff6a410db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:00.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.747+0000 7ff6a94f1700 1 --2- 172.21.15.129:0/1434993871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6a4101680 0x7ff6a410db60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00b0a0 tx=0x7ff69c00b3b0 comp rx=0 tx=0).stop 2024-03-11T20:51:00.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.747+0000 7ff6a94f1700 1 -- 172.21.15.129:0/1434993871 shutdown_connections 2024-03-11T20:51:00.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.747+0000 7ff6a94f1700 1 --2- 172.21.15.129:0/1434993871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6a4101680 0x7ff6a410db60 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:00.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.747+0000 7ff6a94f1700 1 --2- 172.21.15.129:0/1434993871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6a4106d20 0x7ff6a4107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:00.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.747+0000 7ff6a94f1700 1 --2- 172.21.15.129:0/1434993871 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff6a4100ce0 0x7ff6a4101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:00.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.747+0000 7ff6a94f1700 1 -- 172.21.15.129:0/1434993871 >> 172.21.15.129:0/1434993871 conn(0x7ff6a4077c10 msgr2=0x7ff6a4078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:00.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.747+0000 7ff6a94f1700 1 -- 172.21.15.129:0/1434993871 shutdown_connections 2024-03-11T20:51:00.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.747+0000 7ff6a94f1700 1 -- 172.21.15.129:0/1434993871 wait complete. 2024-03-11T20:51:00.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.748+0000 7ff6a94f1700 1 Processor -- start 2024-03-11T20:51:00.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.748+0000 7ff6a94f1700 1 -- start start 2024-03-11T20:51:00.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.748+0000 7ff6a94f1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6a4100ce0 0x7ff6a419d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:00.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.748+0000 7ff6a94f1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6a4101680 0x7ff6a419dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:00.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.748+0000 7ff6a94f1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff6a4106d20 0x7ff6a4197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:00.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.748+0000 7ff6a94f1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a41104f0 con 0x7ff6a4100ce0 2024-03-11T20:51:00.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.748+0000 7ff6a94f1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a4110370 con 0x7ff6a4101680 2024-03-11T20:51:00.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.748+0000 7ff6a94f1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a4110670 con 0x7ff6a4106d20 2024-03-11T20:51:00.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff6a2ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6a4100ce0 0x7ff6a419d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:00.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff6a37fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff6a4106d20 0x7ff6a4197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:00.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff6a2ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6a4100ce0 0x7ff6a419d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42506/0 (socket says 172.21.15.129:42506) 2024-03-11T20:51:00.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff6a2ffd700 1 -- 172.21.15.129:0/1955870373 learned_addr learned my addr 172.21.15.129:0/1955870373 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:51:00.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff6a27fc700 1 --2- 172.21.15.129:0/1955870373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6a4101680 0x7ff6a419dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:00.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff6a37fe700 1 -- 172.21.15.129:0/1955870373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6a4101680 msgr2=0x7ff6a419dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:00.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff6a37fe700 1 --2- 172.21.15.129:0/1955870373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6a4101680 0x7ff6a419dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:00.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff6a37fe700 1 -- 172.21.15.129:0/1955870373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6a4100ce0 msgr2=0x7ff6a419d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:00.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff6a37fe700 1 --2- 172.21.15.129:0/1955870373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6a4100ce0 0x7ff6a419d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:00.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff6a37fe700 1 -- 172.21.15.129:0/1955870373 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a4197fe0 con 0x7ff6a4106d20 2024-03-11T20:51:00.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff6a2ffd700 1 --2- 172.21.15.129:0/1955870373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6a4100ce0 0x7ff6a419d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:51:00.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff6a37fe700 1 --2- 172.21.15.129:0/1955870373 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff6a4106d20 0x7ff6a4197880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00bfd0 tx=0x7ff69c007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:00.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff68bfff700 1 -- 172.21.15.129:0/1955870373 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69c021020 con 0x7ff6a4106d20 2024-03-11T20:51:00.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff6a94f1700 1 -- 172.21.15.129:0/1955870373 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a4198270 con 0x7ff6a4106d20 2024-03-11T20:51:00.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff68bfff700 1 -- 172.21.15.129:0/1955870373 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ff69c004c50 con 0x7ff6a4106d20 2024-03-11T20:51:00.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff68bfff700 1 -- 172.21.15.129:0/1955870373 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69c005050 con 0x7ff6a4106d20 2024-03-11T20:51:00.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.749+0000 7ff6a94f1700 1 -- 172.21.15.129:0/1955870373 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a41a4590 con 0x7ff6a4106d20 2024-03-11T20:51:00.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.750+0000 7ff6a94f1700 1 -- 172.21.15.129:0/1955870373 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6a4068aa0 con 0x7ff6a4106d20 2024-03-11T20:51:00.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.751+0000 7ff68bfff700 1 -- 172.21.15.129:0/1955870373 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7ff69c024020 con 0x7ff6a4106d20 2024-03-11T20:51:00.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.751+0000 7ff68bfff700 1 --2- 172.21.15.129:0/1955870373 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff68c06dd90 0x7ff68c070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:00.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.751+0000 7ff68bfff700 1 -- 172.21.15.129:0/1955870373 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5287+0+0 (secure 0 0 0) 0x7ff69c00f020 con 0x7ff6a4106d20 2024-03-11T20:51:00.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.751+0000 7ff6a2ffd700 1 --2- 172.21.15.129:0/1955870373 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff68c06dd90 0x7ff68c070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:00.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.752+0000 7ff6a2ffd700 1 --2- 172.21.15.129:0/1955870373 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff68c06dd90 0x7ff68c070250 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff6a41989c0 tx=0x7ff69400b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:00.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.756+0000 7ff68bfff700 1 -- 172.21.15.129:0/1955870373 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7ff69c05c780 con 0x7ff6a4106d20 2024-03-11T20:51:00.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.949+0000 7ff6a94f1700 1 -- 172.21.15.129:0/1955870373 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6a41a4760 con 0x7ff6a4106d20 2024-03-11T20:51:00.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.950+0000 7ff68bfff700 1 -- 172.21.15.129:0/1955870373 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+13634 (secure 0 0 0) 0x7ff69c05c080 con 0x7ff6a4106d20 2024-03-11T20:51:00.945 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:51:00.946 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":57,"fsid":"27bbebfa-dfe8-11ee-95c8-87774f69a715","created":"2024-03-11T20:45:35.170893+0000","modified":"2024-03-11T20:51:00.479279+0000","last_up_change":"2024-03-11T20:50:57.449832+0000","last_in_change":"2024-03-11T20:50:39.277288+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T20:48:49.904657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5dbe3ee2-8c30-4242-baf0-a36e1ac0be8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3618360833},{"type":"v1","addr":"172.21.15.129:6803","nonce":3618360833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3618360833},{"type":"v1","addr":"172.21.15.129:6805","nonce":3618360833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":3618360833},{"type":"v1","addr":"172.21.15.129:6809","nonce":3618360833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3618360833},{"type":"v1","addr":"172.21.15.129:6807","nonce":3618360833}]},"public_addr":"172.21.15.129:6803/3618360833","cluster_addr":"172.21.15.129:6805/3618360833","heartbeat_back_addr":"172.21.15.129:6809/3618360833","heartbeat_front_addr":"172.21.15.129:6807/3618360833","state":["exists","up"]},{"osd":1,"uuid":"d5495628-e325-4a58-b682-50a26ef780fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2519234527},{"type":"v1","addr":"172.21.15.129:6811","nonce":2519234527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2519234527},{"type":"v1","addr":"172.21.15.129:6813","nonce":2519234527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":2519234527},{"type":"v1","addr":"172.21.15.129:6817","nonce":2519234527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2519234527},{"type":"v1","addr":"172.21.15.129:6815","nonce":2519234527}]},"public_addr":"172.21.15.129:6811/2519234527","cluster_addr":"172.21.15.129:6813/2519234527","heartbeat_back_addr":"172.21.15.129:6817/2519234527","heartbeat_front_addr":"172.21.15.129:6815/2519234527","state":["exists","up"]},{"osd":2,"uuid":"2b6db120-1132-4369-9f17-777545ad4a07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1600233087},{"type":"v1","addr":"172.21.15.129:6819","nonce":1600233087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1600233087},{"type":"v1","addr":"172.21.15.129:6821","nonce":1600233087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1600233087},{"type":"v1","addr":"172.21.15.129:6825","nonce":1600233087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1600233087},{"type":"v1","addr":"172.21.15.129:6823","nonce":1600233087}]},"public_addr":"172.21.15.129:6819/1600233087","cluster_addr":"172.21.15.129:6821/1600233087","heartbeat_back_addr":"172.21.15.129:6825/1600233087","heartbeat_front_addr":"172.21.15.129:6823/1600233087","state":["exists","up"]},{"osd":3,"uuid":"d9d3db4d-385d-4745-b71f-0796ff551f4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2265856698},{"type":"v1","addr":"172.21.15.129:6827","nonce":2265856698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2265856698},{"type":"v1","addr":"172.21.15.129:6829","nonce":2265856698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2265856698},{"type":"v1","addr":"172.21.15.129:6833","nonce":2265856698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2265856698},{"type":"v1","addr":"172.21.15.129:6831","nonce":2265856698}]},"public_addr":"172.21.15.129:6827/2265856698","cluster_addr":"172.21.15.129:6829/2265856698","heartbeat_back_addr":"172.21.15.129:6833/2265856698","heartbeat_front_addr":"172.21.15.129:6831/2265856698","state":["exists","up"]},{"osd":4,"uuid":"2aa92e2f-5687-4260-985a-3006da3cc803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3991533346},{"type":"v1","addr":"172.21.15.157:6801","nonce":3991533346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3991533346},{"type":"v1","addr":"172.21.15.157:6803","nonce":3991533346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3991533346},{"type":"v1","addr":"172.21.15.157:6807","nonce":3991533346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3991533346},{"type":"v1","addr":"172.21.15.157:6805","nonce":3991533346}]},"public_addr":"172.21.15.157:6801/3991533346","cluster_addr":"172.21.15.157:6803/3991533346","heartbeat_back_addr":"172.21.15.157:6807/3991533346","heartbeat_front_addr":"172.21.15.157:6805/3991533346","state":["exists","up"]},{"osd":5,"uuid":"dac5fa60-f01b-4323-8596-cc0f3a5030c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":2444017632},{"type":"v1","addr":"172.21.15.157:6809","nonce":2444017632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":2444017632},{"type":"v1","addr":"172.21.15.157:6811","nonce":2444017632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":2444017632},{"type":"v1","addr":"172.21.15.157:6815","nonce":2444017632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":2444017632},{"type":"v1","addr":"172.21.15.157:6813","nonce":2444017632}]},"public_addr":"172.21.15.157:6809/2444017632","cluster_addr":"172.21.15.157:6811/2444017632","heartbeat_back_addr":"172.21.15.157:6815/2444017632","heartbeat_front_addr":"172.21.15.157:6813/2444017632","state":["exists","up"]},{"osd":6,"uuid":"9c489abd-5f6c-4fb9-954a-aabd409fffd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":2710731257},{"type":"v1","addr":"172.21.15.157:6817","nonce":2710731257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":2710731257},{"type":"v1","addr":"172.21.15.157:6819","nonce":2710731257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":2710731257},{"type":"v1","addr":"172.21.15.157:6823","nonce":2710731257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":2710731257},{"type":"v1","addr":"172.21.15.157:6821","nonce":2710731257}]},"public_addr":"172.21.15.157:6817/2710731257","cluster_addr":"172.21.15.157:6819/2710731257","heartbeat_back_addr":"172.21.15.157:6823/2710731257","heartbeat_front_addr":"172.21.15.157:6821/2710731257","state":["exists","up"]},{"osd":7,"uuid":"22bacf1f-35c5-4dee-81a6-92dff2f076c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1786686696},{"type":"v1","addr":"172.21.15.157:6825","nonce":1786686696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1786686696},{"type":"v1","addr":"172.21.15.157:6827","nonce":1786686696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1786686696},{"type":"v1","addr":"172.21.15.157:6831","nonce":1786686696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1786686696},{"type":"v1","addr":"172.21.15.157:6829","nonce":1786686696}]},"public_addr":"172.21.15.157:6825/1786686696","cluster_addr":"172.21.15.157:6827/1786686696","heartbeat_back_addr":"172.21.15.157:6831/1786686696","heartbeat_front_addr":"172.21.15.157:6829/1786686696","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:47:54.931014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:48:20.791637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:48:46.661578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:49:13.177044+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:49:37.603176+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:50:03.465298+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:50:29.681381+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:50:55.559486+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2734363055":"2024-03-12T20:46:29.573760+0000","172.21.15.129:0/1185914521":"2024-03-12T20:46:29.573760+0000","172.21.15.129:6800/2213467472":"2024-03-12T20:46:02.671885+0000","172.21.15.129:0/1229097270":"2024-03-12T20:46:02.671885+0000","172.21.15.129:6801/521270824":"2024-03-12T20:46:29.573760+0000","172.21.15.129:6801/2213467472":"2024-03-12T20:46:02.671885+0000","172.21.15.129:0/160155772":"2024-03-12T20:46:29.573760+0000","172.21.15.129:0/4048407456":"2024-03-12T20:46:02.671885+0000","172.21.15.129:0/1059875431":"2024-03-12T20:46:02.671885+0000","172.21.15.129:6800/521270824":"2024-03-12T20:46:29.573760+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T20:51:00.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.952+0000 7ff6a94f1700 1 -- 172.21.15.129:0/1955870373 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff68c06dd90 msgr2=0x7ff68c070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:00.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.952+0000 7ff6a94f1700 1 --2- 172.21.15.129:0/1955870373 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff68c06dd90 0x7ff68c070250 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff6a41989c0 tx=0x7ff69400b500 comp rx=0 tx=0).stop 2024-03-11T20:51:00.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.952+0000 7ff6a94f1700 1 -- 172.21.15.129:0/1955870373 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff6a4106d20 msgr2=0x7ff6a4197880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:00.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.952+0000 7ff6a94f1700 1 --2- 172.21.15.129:0/1955870373 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff6a4106d20 0x7ff6a4197880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00bfd0 tx=0x7ff69c007a20 comp rx=0 tx=0).stop 2024-03-11T20:51:00.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.952+0000 7ff6a94f1700 1 -- 172.21.15.129:0/1955870373 shutdown_connections 2024-03-11T20:51:00.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.952+0000 7ff6a94f1700 1 --2- 172.21.15.129:0/1955870373 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7ff68c06dd90 0x7ff68c070250 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:00.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.952+0000 7ff6a94f1700 1 --2- 172.21.15.129:0/1955870373 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ff6a4106d20 0x7ff6a4197880 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:00.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.952+0000 7ff6a94f1700 1 --2- 172.21.15.129:0/1955870373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6a4100ce0 0x7ff6a419d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:00.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.952+0000 7ff6a94f1700 1 --2- 172.21.15.129:0/1955870373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6a4101680 0x7ff6a419dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:00.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.952+0000 7ff6a94f1700 1 -- 172.21.15.129:0/1955870373 >> 172.21.15.129:0/1955870373 conn(0x7ff6a4077c10 msgr2=0x7ff6a4100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:00.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.952+0000 7ff6a94f1700 1 -- 172.21.15.129:0/1955870373 shutdown_connections 2024-03-11T20:51:00.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:00.952+0000 7ff6a94f1700 1 -- 172.21.15.129:0/1955870373 wait complete. 2024-03-11T20:51:01.522 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:01 smithi129 ceph-mon[30184]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T20:51:01.522 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1955870373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:51:01.522 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:01 smithi129 ceph-mon[42066]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T20:51:01.522 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1955870373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:51:01.552 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T20:48:49.904657+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T20:51:01.553 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T20:51:01.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:01 smithi157 ceph-mon[33359]: osdmap e57: 8 total, 8 up, 8 in 2024-03-11T20:51:01.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1955870373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:51:01.790 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:51:02.603 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:02 smithi129 ceph-mon[30184]: osdmap e58: 8 total, 8 up, 8 in 2024-03-11T20:51:02.603 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:02 smithi129 ceph-mon[30184]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:51:02.603 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:02 smithi129 ceph-mon[42066]: osdmap e58: 8 total, 8 up, 8 in 2024-03-11T20:51:02.603 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:02 smithi129 ceph-mon[42066]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:51:02.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:02 smithi157 ceph-mon[33359]: osdmap e58: 8 total, 8 up, 8 in 2024-03-11T20:51:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:02 smithi157 ceph-mon[33359]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:51:03.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.425+0000 7f0fc9778700 1 -- 172.21.15.129:0/2309699825 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0fc4069a90 msgr2=0x7f0fc4069e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:03.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.425+0000 7f0fc9778700 1 --2- 172.21.15.129:0/2309699825 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0fc4069a90 0x7f0fc4069e70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40098f0 tx=0x7f0fb4009c00 comp rx=0 tx=0).stop 2024-03-11T20:51:03.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.425+0000 7f0fc9778700 1 -- 172.21.15.129:0/2309699825 shutdown_connections 2024-03-11T20:51:03.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.425+0000 7f0fc9778700 1 --2- 172.21.15.129:0/2309699825 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fc410e890 0x7f0fc4110c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:03.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.425+0000 7f0fc9778700 1 --2- 172.21.15.129:0/2309699825 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0fc4069a90 0x7f0fc4069e70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:03.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.425+0000 7f0fc9778700 1 --2- 172.21.15.129:0/2309699825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fc406a3b0 0x7f0fc410e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:03.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.425+0000 7f0fc9778700 1 -- 172.21.15.129:0/2309699825 >> 172.21.15.129:0/2309699825 conn(0x7f0fc4077c10 msgr2=0x7f0fc4078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:03.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.425+0000 7f0fc9778700 1 -- 172.21.15.129:0/2309699825 shutdown_connections 2024-03-11T20:51:03.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.425+0000 7f0fc9778700 1 -- 172.21.15.129:0/2309699825 wait complete. 2024-03-11T20:51:03.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.425+0000 7f0fc9778700 1 Processor -- start 2024-03-11T20:51:03.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc9778700 1 -- start start 2024-03-11T20:51:03.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc9778700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fc4069a90 0x7f0fc4199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:03.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc9778700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fc406a3b0 0x7f0fc419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:03.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc9778700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0fc410e890 0x7f0fc419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:03.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc9778700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc4113730 con 0x7f0fc4069a90 2024-03-11T20:51:03.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc9778700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc41135b0 con 0x7f0fc406a3b0 2024-03-11T20:51:03.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc9778700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc41138b0 con 0x7f0fc410e890 2024-03-11T20:51:03.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc37fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0fc410e890 0x7f0fc419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:03.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc37fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0fc410e890 0x7f0fc419e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:56830/0 (socket says 172.21.15.129:56830) 2024-03-11T20:51:03.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc37fe700 1 -- 172.21.15.129:0/2936350188 learned_addr learned my addr 172.21.15.129:0/2936350188 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:51:03.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc2ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fc4069a90 0x7f0fc4199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:03.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc27fc700 1 --2- 172.21.15.129:0/2936350188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fc406a3b0 0x7f0fc419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:03.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc37fe700 1 -- 172.21.15.129:0/2936350188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fc406a3b0 msgr2=0x7f0fc419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:03.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc37fe700 1 --2- 172.21.15.129:0/2936350188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fc406a3b0 0x7f0fc419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:03.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc37fe700 1 -- 172.21.15.129:0/2936350188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fc4069a90 msgr2=0x7f0fc4199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:03.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc37fe700 1 --2- 172.21.15.129:0/2936350188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fc4069a90 0x7f0fc4199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:03.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc37fe700 1 -- 172.21.15.129:0/2936350188 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc419ed30 con 0x7f0fc410e890 2024-03-11T20:51:03.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc2ffd700 1 --2- 172.21.15.129:0/2936350188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fc4069a90 0x7f0fc4199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:51:03.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.426+0000 7f0fc37fe700 1 --2- 172.21.15.129:0/2936350188 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0fc410e890 0x7f0fc419e5f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0094f0 tx=0x7f0fbc00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:03.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.427+0000 7f0fabfff700 1 -- 172.21.15.129:0/2936350188 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc004db0 con 0x7f0fc410e890 2024-03-11T20:51:03.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.427+0000 7f0fabfff700 1 -- 172.21.15.129:0/2936350188 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0fbc009e20 con 0x7f0fc410e890 2024-03-11T20:51:03.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.427+0000 7f0fc9778700 1 -- 172.21.15.129:0/2936350188 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc419f020 con 0x7f0fc410e890 2024-03-11T20:51:03.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.427+0000 7f0fabfff700 1 -- 172.21.15.129:0/2936350188 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc005a60 con 0x7f0fc410e890 2024-03-11T20:51:03.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.427+0000 7f0fc9778700 1 -- 172.21.15.129:0/2936350188 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc41a68a0 con 0x7f0fc410e890 2024-03-11T20:51:03.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.428+0000 7f0fc9778700 1 -- 172.21.15.129:0/2936350188 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc4068aa0 con 0x7f0fc410e890 2024-03-11T20:51:03.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.428+0000 7f0fabfff700 1 -- 172.21.15.129:0/2936350188 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7f0fbc00db20 con 0x7f0fc410e890 2024-03-11T20:51:03.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.428+0000 7f0fabfff700 1 --2- 172.21.15.129:0/2936350188 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f0fac06dd90 0x7f0fac070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:03.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.429+0000 7f0fabfff700 1 -- 172.21.15.129:0/2936350188 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f0fbc08d1d0 con 0x7f0fc410e890 2024-03-11T20:51:03.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.429+0000 7f0fc2ffd700 1 --2- 172.21.15.129:0/2936350188 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f0fac06dd90 0x7f0fac070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:03.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.429+0000 7f0fc2ffd700 1 --2- 172.21.15.129:0/2936350188 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f0fac06dd90 0x7f0fac070250 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4009fd0 tx=0x7f0fb4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:03.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.433+0000 7f0fabfff700 1 -- 172.21.15.129:0/2936350188 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f0fbc010070 con 0x7f0fc410e890 2024-03-11T20:51:03.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.625+0000 7f0fc9778700 1 -- 172.21.15.129:0/2936350188 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0fc41a6a70 con 0x7f0fc410e890 2024-03-11T20:51:03.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.625+0000 7f0fabfff700 1 -- 172.21.15.129:0/2936350188 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0fbc05af60 con 0x7f0fc410e890 2024-03-11T20:51:03.621 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-03-11T20:51:03.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.627+0000 7f0fc9778700 1 -- 172.21.15.129:0/2936350188 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f0fac06dd90 msgr2=0x7f0fac070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:03.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.627+0000 7f0fc9778700 1 --2- 172.21.15.129:0/2936350188 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f0fac06dd90 0x7f0fac070250 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4009fd0 tx=0x7f0fb4019040 comp rx=0 tx=0).stop 2024-03-11T20:51:03.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.627+0000 7f0fc9778700 1 -- 172.21.15.129:0/2936350188 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0fc410e890 msgr2=0x7f0fc419e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:03.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.627+0000 7f0fc9778700 1 --2- 172.21.15.129:0/2936350188 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0fc410e890 0x7f0fc419e5f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0094f0 tx=0x7f0fbc00d3b0 comp rx=0 tx=0).stop 2024-03-11T20:51:03.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.627+0000 7f0fc9778700 1 -- 172.21.15.129:0/2936350188 shutdown_connections 2024-03-11T20:51:03.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.627+0000 7f0fc9778700 1 --2- 172.21.15.129:0/2936350188 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f0fac06dd90 0x7f0fac070250 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:03.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.627+0000 7f0fc9778700 1 --2- 172.21.15.129:0/2936350188 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0fc410e890 0x7f0fc419e5f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:03.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.627+0000 7f0fc9778700 1 --2- 172.21.15.129:0/2936350188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0fc4069a90 0x7f0fc4199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:03.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.627+0000 7f0fc9778700 1 --2- 172.21.15.129:0/2936350188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0fc406a3b0 0x7f0fc419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:03.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.627+0000 7f0fc9778700 1 -- 172.21.15.129:0/2936350188 >> 172.21.15.129:0/2936350188 conn(0x7f0fc4077c10 msgr2=0x7f0fc40ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:03.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.627+0000 7f0fc9778700 1 -- 172.21.15.129:0/2936350188 shutdown_connections 2024-03-11T20:51:03.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:03.627+0000 7f0fc9778700 1 -- 172.21.15.129:0/2936350188 wait complete. 2024-03-11T20:51:03.918 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2936350188' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T20:51:03.918 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2936350188' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T20:51:03.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2936350188' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T20:51:04.353 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi129 2024-03-11T20:51:04.353 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi129=foo.a' 2024-03-11T20:51:04.603 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:51:04.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:04 smithi129 ceph-mon[30184]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-11T20:51:04.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:04 smithi129 ceph-mon[42066]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-11T20:51:04.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:04 smithi157 ceph-mon[33359]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-11T20:51:06.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.304+0000 7f4892c4a700 1 -- 172.21.15.157:0/549544843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f488c105260 msgr2=0x7f488c107650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:06.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.304+0000 7f4892c4a700 1 --2- 172.21.15.157:0/549544843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f488c105260 0x7f488c107650 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4878009990 tx=0x7f4878009ca0 comp rx=0 tx=0).stop 2024-03-11T20:51:06.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.304+0000 7f4892c4a700 1 -- 172.21.15.157:0/549544843 shutdown_connections 2024-03-11T20:51:06.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.304+0000 7f4892c4a700 1 --2- 172.21.15.157:0/549544843 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f488c107b90 0x7f488c109f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:06.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.304+0000 7f4892c4a700 1 --2- 172.21.15.157:0/549544843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f488c102930 0x7f488c104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:06.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.304+0000 7f4892c4a700 1 --2- 172.21.15.157:0/549544843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f488c105260 0x7f488c107650 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:06.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.304+0000 7f4892c4a700 1 -- 172.21.15.157:0/549544843 >> 172.21.15.157:0/549544843 conn(0x7f488c0fc2a0 msgr2=0x7f488c0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:06.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.304+0000 7f4892c4a700 1 -- 172.21.15.157:0/549544843 shutdown_connections 2024-03-11T20:51:06.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.304+0000 7f4892c4a700 1 -- 172.21.15.157:0/549544843 wait complete. 2024-03-11T20:51:06.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f4892c4a700 1 Processor -- start 2024-03-11T20:51:06.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f4892c4a700 1 -- start start 2024-03-11T20:51:06.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f4892c4a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f488c102930 0x7f488c197540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:06.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f4892c4a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f488c105260 0x7f488c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:06.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f4892c4a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f488c107b90 0x7f488c19eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:06.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f4892c4a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c10ce30 con 0x7f488c102930 2024-03-11T20:51:06.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f4892c4a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c10ccb0 con 0x7f488c107b90 2024-03-11T20:51:06.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f4892c4a700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488c10cfb0 con 0x7f488c105260 2024-03-11T20:51:06.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f48911e7700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f488c107b90 0x7f488c19eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:06.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f48911e7700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f488c107b90 0x7f488c19eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:45424/0 (socket says 172.21.15.157:45424) 2024-03-11T20:51:06.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f48911e7700 1 -- 172.21.15.157:0/1961168354 learned_addr learned my addr 172.21.15.157:0/1961168354 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:51:06.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f48909e6700 1 --2- 172.21.15.157:0/1961168354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f488c102930 0x7f488c197540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:06.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f488bfff700 1 --2- 172.21.15.157:0/1961168354 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f488c105260 0x7f488c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:06.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f48911e7700 1 -- 172.21.15.157:0/1961168354 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f488c105260 msgr2=0x7f488c197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:06.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f48911e7700 1 --2- 172.21.15.157:0/1961168354 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f488c105260 0x7f488c197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:06.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f48911e7700 1 -- 172.21.15.157:0/1961168354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f488c102930 msgr2=0x7f488c197540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:06.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f48911e7700 1 --2- 172.21.15.157:0/1961168354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f488c102930 0x7f488c197540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:06.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f48911e7700 1 -- 172.21.15.157:0/1961168354 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c19f1a0 con 0x7f488c107b90 2024-03-11T20:51:06.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f48911e7700 1 --2- 172.21.15.157:0/1961168354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f488c107b90 0x7f488c19eb00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4884007ec0 tx=0x7f488400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:06.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.305+0000 7f4889ffb700 1 -- 172.21.15.157:0/1961168354 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488400dd70 con 0x7f488c107b90 2024-03-11T20:51:06.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.306+0000 7f4892c4a700 1 -- 172.21.15.157:0/1961168354 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488c19f490 con 0x7f488c107b90 2024-03-11T20:51:06.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.306+0000 7f4889ffb700 1 -- 172.21.15.157:0/1961168354 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4884018070 con 0x7f488c107b90 2024-03-11T20:51:06.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.306+0000 7f4889ffb700 1 -- 172.21.15.157:0/1961168354 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4884012690 con 0x7f488c107b90 2024-03-11T20:51:06.305 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.306+0000 7f4892c4a700 1 -- 172.21.15.157:0/1961168354 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488c19f9d0 con 0x7f488c107b90 2024-03-11T20:51:06.305 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.307+0000 7f4892c4a700 1 -- 172.21.15.157:0/1961168354 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488c068aa0 con 0x7f488c107b90 2024-03-11T20:51:06.305 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.307+0000 7f4889ffb700 1 -- 172.21.15.157:0/1961168354 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7f48840133f0 con 0x7f488c107b90 2024-03-11T20:51:06.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.308+0000 7f4889ffb700 1 --2- 172.21.15.157:0/1961168354 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f487c06dd90 0x7f487c070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:06.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.308+0000 7f4889ffb700 1 -- 172.21.15.157:0/1961168354 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f488408d6a0 con 0x7f488c107b90 2024-03-11T20:51:06.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.308+0000 7f48909e6700 1 --2- 172.21.15.157:0/1961168354 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f487c06dd90 0x7f487c070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:06.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.309+0000 7f48909e6700 1 --2- 172.21.15.157:0/1961168354 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f487c06dd90 0x7f487c070250 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f488000bbd0 tx=0x7f488000b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:06.308 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.313+0000 7f4889ffb700 1 -- 172.21.15.157:0/1961168354 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f4884010070 con 0x7f488c107b90 2024-03-11T20:51:06.513 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.517+0000 7f4892c4a700 1 -- 172.21.15.157:0/1961168354 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi129=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f488c02daf0 con 0x7f487c06dd90 2024-03-11T20:51:06.520 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.524+0000 7f4889ffb700 1 -- 172.21.15.157:0/1961168354 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (secure 0 0 0) 0x7f488c02daf0 con 0x7f487c06dd90 2024-03-11T20:51:06.520 INFO:teuthology.orchestra.run.smithi157.stdout:Scheduled rgw.foo.a update... 2024-03-11T20:51:06.522 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.526+0000 7f4892c4a700 1 -- 172.21.15.157:0/1961168354 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f487c06dd90 msgr2=0x7f487c070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:06.523 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.526+0000 7f4892c4a700 1 --2- 172.21.15.157:0/1961168354 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f487c06dd90 0x7f487c070250 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f488000bbd0 tx=0x7f488000b5b0 comp rx=0 tx=0).stop 2024-03-11T20:51:06.523 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.526+0000 7f4892c4a700 1 -- 172.21.15.157:0/1961168354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f488c107b90 msgr2=0x7f488c19eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:06.523 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.526+0000 7f4892c4a700 1 --2- 172.21.15.157:0/1961168354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f488c107b90 0x7f488c19eb00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4884007ec0 tx=0x7f488400d3b0 comp rx=0 tx=0).stop 2024-03-11T20:51:06.523 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.527+0000 7f4892c4a700 1 -- 172.21.15.157:0/1961168354 shutdown_connections 2024-03-11T20:51:06.523 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.527+0000 7f4892c4a700 1 --2- 172.21.15.157:0/1961168354 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f487c06dd90 0x7f487c070250 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:06.523 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.527+0000 7f4892c4a700 1 --2- 172.21.15.157:0/1961168354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f488c107b90 0x7f488c19eb00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:06.523 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.527+0000 7f4892c4a700 1 --2- 172.21.15.157:0/1961168354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f488c102930 0x7f488c197540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:06.523 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.527+0000 7f4892c4a700 1 --2- 172.21.15.157:0/1961168354 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f488c105260 0x7f488c197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:06.523 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.527+0000 7f4892c4a700 1 -- 172.21.15.157:0/1961168354 >> 172.21.15.157:0/1961168354 conn(0x7f488c0fc2a0 msgr2=0x7f488c1035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:06.523 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.527+0000 7f4892c4a700 1 -- 172.21.15.157:0/1961168354 shutdown_connections 2024-03-11T20:51:06.523 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:06.527+0000 7f4892c4a700 1 -- 172.21.15.157:0/1961168354 wait complete. 2024-03-11T20:51:06.827 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:06 smithi157 ceph-mon[33359]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-11T20:51:06.827 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:06 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:06.827 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:06 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:51:06.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:06 smithi129 ceph-mon[30184]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-11T20:51:06.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:06 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:06.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:06 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:51:06.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:06 smithi129 ceph-mon[42066]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-11T20:51:06.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:06 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:06.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:06 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:51:07.124 DEBUG:teuthology.orchestra.run.smithi129:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@rgw.foo.a.service 2024-03-11T20:51:07.127 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi157 2024-03-11T20:51:07.127 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-11T20:51:07.368 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:51:07.823 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:07 smithi157 ceph-mon[33359]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi129=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:07.823 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:07 smithi157 ceph-mon[33359]: Saving service rgw.foo.a spec with placement smithi129=foo.a;count:1 2024-03-11T20:51:07.823 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:07 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:07.823 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:07 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:51:07.823 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:07 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:07.823 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:07 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T20:51:07.823 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:07 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T20:51:07.823 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:07 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:07.823 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:07 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:07.940 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[42066]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi129=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:07.940 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[42066]: Saving service rgw.foo.a spec with placement smithi129=foo.a;count:1 2024-03-11T20:51:07.940 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:07.940 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:51:07.940 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:07.940 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T20:51:07.940 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T20:51:07.940 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:07.940 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:07.941 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[30184]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi129=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:07.941 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[30184]: Saving service rgw.foo.a spec with placement smithi129=foo.a;count:1 2024-03-11T20:51:07.941 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:07.941 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:51:07.941 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:07.941 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T20:51:07.941 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T20:51:07.941 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:07.941 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:07 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:08.821 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:08 smithi157 ceph-mon[33359]: Deploying daemon rgw.foo.a on smithi129 2024-03-11T20:51:08.821 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:08 smithi157 ceph-mon[33359]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T20:51:08.893 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:08 smithi129 ceph-mon[30184]: Deploying daemon rgw.foo.a on smithi129 2024-03-11T20:51:08.893 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:08 smithi129 ceph-mon[30184]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T20:51:08.893 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:08 smithi129 ceph-mon[42066]: Deploying daemon rgw.foo.a on smithi129 2024-03-11T20:51:08.894 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:08 smithi129 ceph-mon[42066]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T20:51:08.929 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.932+0000 7fc45a961700 1 -- 172.21.15.157:0/269443292 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc454102080 msgr2=0x7fc454108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:08.929 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.932+0000 7fc45a961700 1 --2- 172.21.15.157:0/269443292 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc454102080 0x7fc454108930 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc44c00c240 tx=0x7fc44c00c550 comp rx=0 tx=0).stop 2024-03-11T20:51:08.929 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.932+0000 7fc45a961700 1 -- 172.21.15.157:0/269443292 shutdown_connections 2024-03-11T20:51:08.929 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.932+0000 7fc45a961700 1 --2- 172.21.15.157:0/269443292 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc454102080 0x7fc454108930 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:08.929 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.932+0000 7fc45a961700 1 --2- 172.21.15.157:0/269443292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4541004e0 0x7fc454100900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:08.929 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.932+0000 7fc45a961700 1 --2- 172.21.15.157:0/269443292 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc4541016e0 0x7fc454101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:08.929 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.932+0000 7fc45a961700 1 -- 172.21.15.157:0/269443292 >> 172.21.15.157:0/269443292 conn(0x7fc4540fba80 msgr2=0x7fc4540fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:08.929 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.932+0000 7fc45a961700 1 -- 172.21.15.157:0/269443292 shutdown_connections 2024-03-11T20:51:08.929 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.932+0000 7fc45a961700 1 -- 172.21.15.157:0/269443292 wait complete. 2024-03-11T20:51:08.929 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.933+0000 7fc45a961700 1 Processor -- start 2024-03-11T20:51:08.929 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.933+0000 7fc45a961700 1 -- start start 2024-03-11T20:51:08.931 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.933+0000 7fc45a961700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4541004e0 0x7fc454197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:08.931 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.933+0000 7fc45a961700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc4541016e0 0x7fc454197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:08.931 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.933+0000 7fc45a961700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc454102080 0x7fc45419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:08.931 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.933+0000 7fc45a961700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45410ad80 con 0x7fc4541004e0 2024-03-11T20:51:08.932 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.933+0000 7fc45a961700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45410ac00 con 0x7fc4541016e0 2024-03-11T20:51:08.932 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.933+0000 7fc45a961700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45410af00 con 0x7fc454102080 2024-03-11T20:51:08.932 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.933+0000 7fc4537fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc4541016e0 0x7fc454197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:08.932 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.933+0000 7fc4537fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc4541016e0 0x7fc454197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:45440/0 (socket says 172.21.15.157:45440) 2024-03-11T20:51:08.932 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.933+0000 7fc4537fe700 1 -- 172.21.15.157:0/1903692530 learned_addr learned my addr 172.21.15.157:0/1903692530 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:51:08.932 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.934+0000 7fc458efe700 1 --2- 172.21.15.157:0/1903692530 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc454102080 0x7fc45419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:08.932 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.934+0000 7fc453fff700 1 --2- 172.21.15.157:0/1903692530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4541004e0 0x7fc454197510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:08.932 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.934+0000 7fc4537fe700 1 -- 172.21.15.157:0/1903692530 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc454102080 msgr2=0x7fc45419ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:08.932 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.934+0000 7fc4537fe700 1 --2- 172.21.15.157:0/1903692530 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc454102080 0x7fc45419ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:08.932 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.934+0000 7fc4537fe700 1 -- 172.21.15.157:0/1903692530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4541004e0 msgr2=0x7fc454197510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:08.932 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.934+0000 7fc4537fe700 1 --2- 172.21.15.157:0/1903692530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4541004e0 0x7fc454197510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:08.932 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.934+0000 7fc4537fe700 1 -- 172.21.15.157:0/1903692530 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45419f170 con 0x7fc4541016e0 2024-03-11T20:51:08.932 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.934+0000 7fc453fff700 1 --2- 172.21.15.157:0/1903692530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4541004e0 0x7fc454197510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T20:51:08.932 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.934+0000 7fc4537fe700 1 --2- 172.21.15.157:0/1903692530 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc4541016e0 0x7fc454197a50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc44800c970 tx=0x7fc44800cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:08.932 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.934+0000 7fc4517fa700 1 -- 172.21.15.157:0/1903692530 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448007aa0 con 0x7fc4541016e0 2024-03-11T20:51:08.933 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.934+0000 7fc45a961700 1 -- 172.21.15.157:0/1903692530 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45419f460 con 0x7fc4541016e0 2024-03-11T20:51:08.933 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.934+0000 7fc4517fa700 1 -- 172.21.15.157:0/1903692530 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc448004cb0 con 0x7fc4541016e0 2024-03-11T20:51:08.933 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.934+0000 7fc4517fa700 1 -- 172.21.15.157:0/1903692530 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4480054c0 con 0x7fc4541016e0 2024-03-11T20:51:08.933 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.934+0000 7fc45a961700 1 -- 172.21.15.157:0/1903692530 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45419f9a0 con 0x7fc4541016e0 2024-03-11T20:51:08.933 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.935+0000 7fc45a961700 1 -- 172.21.15.157:0/1903692530 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc454068aa0 con 0x7fc4541016e0 2024-03-11T20:51:08.933 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.936+0000 7fc4517fa700 1 -- 172.21.15.157:0/1903692530 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7fc448004750 con 0x7fc4541016e0 2024-03-11T20:51:08.934 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.936+0000 7fc4517fa700 1 --2- 172.21.15.157:0/1903692530 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fc43c06dd90 0x7fc43c070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:08.934 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.936+0000 7fc4517fa700 1 -- 172.21.15.157:0/1903692530 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fc44808d6a0 con 0x7fc4541016e0 2024-03-11T20:51:08.934 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.936+0000 7fc453fff700 1 --2- 172.21.15.157:0/1903692530 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fc43c06dd90 0x7fc43c070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:08.934 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.937+0000 7fc453fff700 1 --2- 172.21.15.157:0/1903692530 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fc43c06dd90 0x7fc43c070250 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc444005fd0 tx=0x7fc444005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:08.936 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:08.941+0000 7fc4517fa700 1 -- 172.21.15.157:0/1903692530 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fc44805b430 con 0x7fc4541016e0 2024-03-11T20:51:09.128 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:09.132+0000 7fc45a961700 1 -- 172.21.15.157:0/1903692530 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7fc45402d830 con 0x7fc4541016e0 2024-03-11T20:51:09.478 INFO:journalctl@ceph.rgw.foo.a.smithi129.stdout:Mar 11 20:51:09 smithi129 systemd[1]: Starting Ceph rgw.foo.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T20:51:09.692 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:09.696+0000 7fc4517fa700 1 -- 172.21.15.157:0/1903692530 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v59) v1 ==== 160+0+0 (secure 0 0 0) 0x7fc44805ad30 con 0x7fc4541016e0 2024-03-11T20:51:09.694 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:09.698+0000 7fc45a961700 1 -- 172.21.15.157:0/1903692530 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fc43c06dd90 msgr2=0x7fc43c070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:09.694 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:09.698+0000 7fc45a961700 1 --2- 172.21.15.157:0/1903692530 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fc43c06dd90 0x7fc43c070250 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc444005fd0 tx=0x7fc444005dc0 comp rx=0 tx=0).stop 2024-03-11T20:51:09.694 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:09.698+0000 7fc45a961700 1 -- 172.21.15.157:0/1903692530 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc4541016e0 msgr2=0x7fc454197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:09.694 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:09.698+0000 7fc45a961700 1 --2- 172.21.15.157:0/1903692530 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc4541016e0 0x7fc454197a50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc44800c970 tx=0x7fc44800cd30 comp rx=0 tx=0).stop 2024-03-11T20:51:09.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:09.698+0000 7fc45a961700 1 -- 172.21.15.157:0/1903692530 shutdown_connections 2024-03-11T20:51:09.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:09.698+0000 7fc45a961700 1 --2- 172.21.15.157:0/1903692530 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fc43c06dd90 0x7fc43c070250 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:09.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:09.698+0000 7fc45a961700 1 --2- 172.21.15.157:0/1903692530 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc454102080 0x7fc45419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:09.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:09.698+0000 7fc45a961700 1 --2- 172.21.15.157:0/1903692530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4541004e0 0x7fc454197510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:09.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:09.698+0000 7fc45a961700 1 --2- 172.21.15.157:0/1903692530 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc4541016e0 0x7fc454197a50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:09.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:09.698+0000 7fc45a961700 1 -- 172.21.15.157:0/1903692530 >> 172.21.15.157:0/1903692530 conn(0x7fc4540fba80 msgr2=0x7fc4541049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:09.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:09.699+0000 7fc45a961700 1 -- 172.21.15.157:0/1903692530 shutdown_connections 2024-03-11T20:51:09.695 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:09.699+0000 7fc45a961700 1 -- 172.21.15.157:0/1903692530 wait complete. 2024-03-11T20:51:09.695 INFO:teuthology.orchestra.run.smithi157.stderr:pool 'datapool' created 2024-03-11T20:51:09.923 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.157:0/1903692530' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T20:51:09.924 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T20:51:09.924 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.157:0/1903692530' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T20:51:09.924 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T20:51:09.924 INFO:journalctl@ceph.rgw.foo.a.smithi129.stdout:Mar 11 20:51:09 smithi129 podman[70142]: 2024-03-11 20:51:09.584733079 +0000 UTC m=+0.111350132 container create 73c7fd587e6b845a91bc80dfdd39573445e430b5a1b1a4eb40cb92755fea84f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-11T20:51:09.924 INFO:journalctl@ceph.rgw.foo.a.smithi129.stdout:Mar 11 20:51:09 smithi129 podman[70142]: 2024-03-11 20:51:09.491306125 +0000 UTC m=+0.017923182 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T20:51:09.925 INFO:journalctl@ceph.rgw.foo.a.smithi129.stdout:Mar 11 20:51:09 smithi129 podman[70142]: 2024-03-11 20:51:09.828932696 +0000 UTC m=+0.355549751 container init 73c7fd587e6b845a91bc80dfdd39573445e430b5a1b1a4eb40cb92755fea84f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-rgw-foo-a, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-11T20:51:09.925 INFO:journalctl@ceph.rgw.foo.a.smithi129.stdout:Mar 11 20:51:09 smithi129 podman[70142]: 2024-03-11 20:51:09.8340987 +0000 UTC m=+0.360715753 container start 73c7fd587e6b845a91bc80dfdd39573445e430b5a1b1a4eb40cb92755fea84f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e) 2024-03-11T20:51:09.925 INFO:journalctl@ceph.rgw.foo.a.smithi129.stdout:Mar 11 20:51:09 smithi129 bash[70142]: 73c7fd587e6b845a91bc80dfdd39573445e430b5a1b1a4eb40cb92755fea84f0 2024-03-11T20:51:09.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.157:0/1903692530' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T20:51:09.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T20:51:10.228 INFO:journalctl@ceph.rgw.foo.a.smithi129.stdout:Mar 11 20:51:09 smithi129 systemd[1]: Started Ceph rgw.foo.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T20:51:10.342 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- rbd pool init datapool 2024-03-11T20:51:10.586 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:51:10.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:10 smithi157 ceph-mon[33359]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-11T20:51:10.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T20:51:10.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:10 smithi157 ceph-mon[33359]: osdmap e59: 8 total, 8 up, 8 in 2024-03-11T20:51:10.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:10 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:10.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:10 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:10.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:10 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:10.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:10 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:10.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:10 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:10.953 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:10 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:51:10.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[42066]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-11T20:51:10.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T20:51:10.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[42066]: osdmap e59: 8 total, 8 up, 8 in 2024-03-11T20:51:10.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:10.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:10.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:10.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:10.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:10.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:51:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[30184]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-11T20:51:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T20:51:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[30184]: osdmap e59: 8 total, 8 up, 8 in 2024-03-11T20:51:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:10 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:51:11.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:11 smithi157 ceph-mon[33359]: Saving service rgw.foo.a spec with placement smithi129=foo.a;count:1 2024-03-11T20:51:11.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:11 smithi157 ceph-mon[33359]: osdmap e60: 8 total, 8 up, 8 in 2024-03-11T20:51:11.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3404719286' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T20:51:11.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:11 smithi129 ceph-mon[30184]: Saving service rgw.foo.a spec with placement smithi129=foo.a;count:1 2024-03-11T20:51:11.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:11 smithi129 ceph-mon[30184]: osdmap e60: 8 total, 8 up, 8 in 2024-03-11T20:51:11.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3404719286' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T20:51:11.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:11 smithi129 ceph-mon[42066]: Saving service rgw.foo.a spec with placement smithi129=foo.a;count:1 2024-03-11T20:51:11.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:11 smithi129 ceph-mon[42066]: osdmap e60: 8 total, 8 up, 8 in 2024-03-11T20:51:11.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3404719286' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T20:51:12.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[30184]: pgmap v191: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:51:12.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[30184]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:51:12.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3404719286' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T20:51:12.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[30184]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T20:51:12.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:12.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:51:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.157:0/61931005' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T20:51:12.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[42066]: pgmap v191: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:51:12.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[42066]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:51:12.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3404719286' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T20:51:12.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[42066]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T20:51:12.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:12.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:12.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:12.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:51:12.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.157:0/61931005' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T20:51:12.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:12 smithi157 ceph-mon[33359]: pgmap v191: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:51:12.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:12 smithi157 ceph-mon[33359]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:51:12.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3404719286' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T20:51:12.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:12 smithi157 ceph-mon[33359]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T20:51:12.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:12 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:12.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:12 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:12.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:12 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:12.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:12 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:51:12.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.157:0/61931005' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T20:51:13.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:13 smithi129 ceph-mon[30184]: Checking dashboard <-> RGW credentials 2024-03-11T20:51:13.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.157:0/61931005' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T20:51:13.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:13 smithi129 ceph-mon[30184]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T20:51:13.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1822711208' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T20:51:13.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T20:51:13.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T20:51:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:13 smithi129 ceph-mon[42066]: Checking dashboard <-> RGW credentials 2024-03-11T20:51:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.157:0/61931005' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T20:51:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:13 smithi129 ceph-mon[42066]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T20:51:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1822711208' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T20:51:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T20:51:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T20:51:13.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:13 smithi157 ceph-mon[33359]: Checking dashboard <-> RGW credentials 2024-03-11T20:51:13.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.157:0/61931005' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T20:51:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:13 smithi157 ceph-mon[33359]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T20:51:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1822711208' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T20:51:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T20:51:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T20:51:14.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:14 smithi129 ceph-mon[30184]: pgmap v194: 68 pgs: 16 active+clean, 6 creating+peering, 46 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-11T20:51:14.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T20:51:14.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T20:51:14.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:14 smithi129 ceph-mon[30184]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T20:51:14.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:14 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:14.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:14 smithi129 ceph-mon[42066]: pgmap v194: 68 pgs: 16 active+clean, 6 creating+peering, 46 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-11T20:51:14.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T20:51:14.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T20:51:14.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:14 smithi129 ceph-mon[42066]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T20:51:14.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:14 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:14.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:14 smithi157 ceph-mon[33359]: pgmap v194: 68 pgs: 16 active+clean, 6 creating+peering, 46 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-11T20:51:14.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T20:51:14.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T20:51:14.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:14 smithi157 ceph-mon[33359]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T20:51:14.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:14 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:15.532 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.157 --placement '1;smithi157=iscsi.a' 2024-03-11T20:51:15.793 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:51:15.915 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:15 smithi157 ceph-mon[33359]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T20:51:15.915 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T20:51:15.915 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1822711208' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T20:51:15.915 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T20:51:15.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:15 smithi129 ceph-mon[30184]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T20:51:15.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T20:51:15.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1822711208' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T20:51:15.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T20:51:15.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:15 smithi129 ceph-mon[42066]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T20:51:15.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T20:51:15.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1822711208' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T20:51:15.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T20:51:16.962 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:16 smithi157 ceph-mon[33359]: pgmap v197: 100 pgs: 33 active+clean, 24 creating+peering, 43 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-11T20:51:16.962 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T20:51:16.962 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:16 smithi157 ceph-mon[33359]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T20:51:16.962 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:16 smithi157 ceph-mon[33359]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T20:51:16.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:16 smithi129 ceph-mon[30184]: pgmap v197: 100 pgs: 33 active+clean, 24 creating+peering, 43 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-11T20:51:16.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T20:51:16.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:16 smithi129 ceph-mon[30184]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T20:51:16.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:16 smithi129 ceph-mon[30184]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T20:51:16.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:16 smithi129 ceph-mon[42066]: pgmap v197: 100 pgs: 33 active+clean, 24 creating+peering, 43 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-11T20:51:16.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T20:51:16.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:16 smithi129 ceph-mon[42066]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T20:51:16.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:16 smithi129 ceph-mon[42066]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T20:51:17.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.453+0000 7fb7fd40d700 1 -- 172.21.15.157:0/92220389 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7f81028d0 msgr2=0x7fb7f8109180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:17.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.453+0000 7fb7fd40d700 1 --2- 172.21.15.157:0/92220389 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7f81028d0 0x7fb7f8109180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000b0d0 tx=0x7fb7f000b3e0 comp rx=0 tx=0).stop 2024-03-11T20:51:17.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.453+0000 7fb7fd40d700 1 -- 172.21.15.157:0/92220389 shutdown_connections 2024-03-11T20:51:17.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.453+0000 7fb7fd40d700 1 --2- 172.21.15.157:0/92220389 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7f81028d0 0x7fb7f8109180 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:17.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.453+0000 7fb7fd40d700 1 --2- 172.21.15.157:0/92220389 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f8100d30 0x7fb7f8101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:17.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.453+0000 7fb7fd40d700 1 --2- 172.21.15.157:0/92220389 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f8101f30 0x7fb7f8102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:17.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.453+0000 7fb7fd40d700 1 -- 172.21.15.157:0/92220389 >> 172.21.15.157:0/92220389 conn(0x7fb7f80fc2d0 msgr2=0x7fb7f80fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:17.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.453+0000 7fb7fd40d700 1 -- 172.21.15.157:0/92220389 shutdown_connections 2024-03-11T20:51:17.451 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.453+0000 7fb7fd40d700 1 -- 172.21.15.157:0/92220389 wait complete. 2024-03-11T20:51:17.451 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.454+0000 7fb7fd40d700 1 Processor -- start 2024-03-11T20:51:17.451 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.454+0000 7fb7fd40d700 1 -- start start 2024-03-11T20:51:17.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.454+0000 7fb7fd40d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f8100d30 0x7fb7f81974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:17.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.454+0000 7fb7fd40d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f8101f30 0x7fb7f8197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:17.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.454+0000 7fb7fd40d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7f81028d0 0x7fb7f819ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:17.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.454+0000 7fb7fd40d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f810be10 con 0x7fb7f8100d30 2024-03-11T20:51:17.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.454+0000 7fb7fd40d700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f810bc90 con 0x7fb7f81028d0 2024-03-11T20:51:17.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.454+0000 7fb7fd40d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f810bf90 con 0x7fb7f8101f30 2024-03-11T20:51:17.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.454+0000 7fb7f77fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7f81028d0 0x7fb7f819ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:17.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.455+0000 7fb7f77fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7f81028d0 0x7fb7f819ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:36886/0 (socket says 172.21.15.157:36886) 2024-03-11T20:51:17.453 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.455+0000 7fb7f77fe700 1 -- 172.21.15.157:0/2024751524 learned_addr learned my addr 172.21.15.157:0/2024751524 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:51:17.454 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.455+0000 7fb7f6ffd700 1 --2- 172.21.15.157:0/2024751524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f8100d30 0x7fb7f81974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:17.454 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.455+0000 7fb7f67fc700 1 --2- 172.21.15.157:0/2024751524 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f8101f30 0x7fb7f8197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:17.454 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.455+0000 7fb7f77fe700 1 -- 172.21.15.157:0/2024751524 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f8101f30 msgr2=0x7fb7f8197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:17.454 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.455+0000 7fb7f77fe700 1 --2- 172.21.15.157:0/2024751524 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f8101f30 0x7fb7f8197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:17.454 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.455+0000 7fb7f77fe700 1 -- 172.21.15.157:0/2024751524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f8100d30 msgr2=0x7fb7f81974f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:17.454 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.455+0000 7fb7f77fe700 1 --2- 172.21.15.157:0/2024751524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f8100d30 0x7fb7f81974f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:17.454 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.455+0000 7fb7f77fe700 1 -- 172.21.15.157:0/2024751524 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f819f100 con 0x7fb7f81028d0 2024-03-11T20:51:17.454 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.455+0000 7fb7f77fe700 1 --2- 172.21.15.157:0/2024751524 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7f81028d0 0x7fb7f819ea60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0007d90 tx=0x7fb7f0009bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:17.454 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.455+0000 7fb7dffff700 1 -- 172.21.15.157:0/2024751524 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f001f020 con 0x7fb7f81028d0 2024-03-11T20:51:17.454 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.455+0000 7fb7dffff700 1 -- 172.21.15.157:0/2024751524 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb7f0003e10 con 0x7fb7f81028d0 2024-03-11T20:51:17.454 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.455+0000 7fb7fd40d700 1 -- 172.21.15.157:0/2024751524 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f819f390 con 0x7fb7f81028d0 2024-03-11T20:51:17.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.455+0000 7fb7dffff700 1 -- 172.21.15.157:0/2024751524 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f001b8d0 con 0x7fb7f81028d0 2024-03-11T20:51:17.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.455+0000 7fb7fd40d700 1 -- 172.21.15.157:0/2024751524 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f819f860 con 0x7fb7f81028d0 2024-03-11T20:51:17.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.456+0000 7fb7fd40d700 1 -- 172.21.15.157:0/2024751524 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f80502f0 con 0x7fb7f81028d0 2024-03-11T20:51:17.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.457+0000 7fb7dffff700 1 -- 172.21.15.157:0/2024751524 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7fb7f0022020 con 0x7fb7f81028d0 2024-03-11T20:51:17.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.457+0000 7fb7dffff700 1 --2- 172.21.15.157:0/2024751524 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb7e006de30 0x7fb7e00702f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:17.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.457+0000 7fb7dffff700 1 -- 172.21.15.157:0/2024751524 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7121+0+0 (secure 0 0 0) 0x7fb7f008f380 con 0x7fb7f81028d0 2024-03-11T20:51:17.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.457+0000 7fb7f6ffd700 1 --2- 172.21.15.157:0/2024751524 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb7e006de30 0x7fb7e00702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:17.455 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.458+0000 7fb7f6ffd700 1 --2- 172.21.15.157:0/2024751524 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb7e006de30 0x7fb7e00702f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8101d90 tx=0x7fb7e8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:17.457 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.462+0000 7fb7dffff700 1 -- 172.21.15.157:0/2024751524 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fb7f0029020 con 0x7fb7f81028d0 2024-03-11T20:51:17.666 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.670+0000 7fb7fd40d700 1 -- 172.21.15.157:0/2024751524 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.157", "placement": "1;smithi157=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7f80fdf10 con 0x7fb7e006de30 2024-03-11T20:51:17.674 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.678+0000 7fb7dffff700 1 -- 172.21.15.157:0/2024751524 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+35 (secure 0 0 0) 0x7fb7f80fdf10 con 0x7fb7e006de30 2024-03-11T20:51:17.674 INFO:teuthology.orchestra.run.smithi157.stdout:Scheduled iscsi.datapool update... 2024-03-11T20:51:17.677 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.681+0000 7fb7fd40d700 1 -- 172.21.15.157:0/2024751524 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb7e006de30 msgr2=0x7fb7e00702f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:17.677 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.681+0000 7fb7fd40d700 1 --2- 172.21.15.157:0/2024751524 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb7e006de30 0x7fb7e00702f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8101d90 tx=0x7fb7e8009190 comp rx=0 tx=0).stop 2024-03-11T20:51:17.677 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.681+0000 7fb7fd40d700 1 -- 172.21.15.157:0/2024751524 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7f81028d0 msgr2=0x7fb7f819ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:17.677 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.681+0000 7fb7fd40d700 1 --2- 172.21.15.157:0/2024751524 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7f81028d0 0x7fb7f819ea60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0007d90 tx=0x7fb7f0009bf0 comp rx=0 tx=0).stop 2024-03-11T20:51:17.677 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.681+0000 7fb7fd40d700 1 -- 172.21.15.157:0/2024751524 shutdown_connections 2024-03-11T20:51:17.677 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.681+0000 7fb7fd40d700 1 --2- 172.21.15.157:0/2024751524 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7fb7e006de30 0x7fb7e00702f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:17.677 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.681+0000 7fb7fd40d700 1 --2- 172.21.15.157:0/2024751524 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7f81028d0 0x7fb7f819ea60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:17.677 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.681+0000 7fb7fd40d700 1 --2- 172.21.15.157:0/2024751524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f8100d30 0x7fb7f81974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:17.677 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.681+0000 7fb7fd40d700 1 --2- 172.21.15.157:0/2024751524 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb7f8101f30 0x7fb7f8197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:17.677 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.681+0000 7fb7fd40d700 1 -- 172.21.15.157:0/2024751524 >> 172.21.15.157:0/2024751524 conn(0x7fb7f80fc2d0 msgr2=0x7fb7f80fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:17.677 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.681+0000 7fb7fd40d700 1 -- 172.21.15.157:0/2024751524 shutdown_connections 2024-03-11T20:51:17.678 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:17.681+0000 7fb7fd40d700 1 -- 172.21.15.157:0/2024751524 wait complete. 2024-03-11T20:51:17.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:17 smithi157 ceph-mon[33359]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T20:51:17.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1822711208' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:51:17.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:51:17.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:51:17.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:17 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:18.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:17 smithi129 ceph-mon[30184]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T20:51:18.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1822711208' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:51:18.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:51:18.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:51:18.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:17 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:18.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:17 smithi129 ceph-mon[42066]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T20:51:18.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1822711208' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:51:18.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:51:18.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T20:51:18.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:17 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:18.321 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-11T20:51:18.321 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:51:18.321 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-11T20:51:18.356 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-11T20:51:18.356 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-11T20:51:18.386 DEBUG:teuthology.orchestra.run.smithi157:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@iscsi.iscsi.a.service 2024-03-11T20:51:18.430 INFO:tasks.cephadm:Adding prometheus.a on smithi157 2024-03-11T20:51:18.430 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi157=a' 2024-03-11T20:51:18.705 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:51:19.001 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:18 smithi157 ceph-mon[33359]: pgmap v200: 132 pgs: 50 active+clean, 34 creating+peering, 48 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 2 op/s 2024-03-11T20:51:19.001 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:18 smithi157 ceph-mon[33359]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.157", "placement": "1;smithi157=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:19.001 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:18 smithi157 ceph-mon[33359]: Saving service iscsi.datapool spec with placement smithi157=iscsi.a;count:1 2024-03-11T20:51:19.001 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T20:51:19.001 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T20:51:19.001 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:18 smithi157 ceph-mon[33359]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T20:51:19.001 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1822711208' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:51:19.002 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:51:19.002 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:51:19.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[30184]: pgmap v200: 132 pgs: 50 active+clean, 34 creating+peering, 48 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 2 op/s 2024-03-11T20:51:19.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[30184]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.157", "placement": "1;smithi157=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:19.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[30184]: Saving service iscsi.datapool spec with placement smithi157=iscsi.a;count:1 2024-03-11T20:51:19.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T20:51:19.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T20:51:19.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[30184]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T20:51:19.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1822711208' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:51:19.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:51:19.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:51:19.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[42066]: pgmap v200: 132 pgs: 50 active+clean, 34 creating+peering, 48 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 2 op/s 2024-03-11T20:51:19.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[42066]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.157", "placement": "1;smithi157=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:19.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[42066]: Saving service iscsi.datapool spec with placement smithi157=iscsi.a;count:1 2024-03-11T20:51:19.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T20:51:19.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T20:51:19.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[42066]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T20:51:19.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1822711208' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:51:19.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:51:19.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T20:51:19.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T20:51:19.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T20:51:19.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:19 smithi157 ceph-mon[33359]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T20:51:19.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:19 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:19.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:19 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:19.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:19 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:19.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:19 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T20:51:19.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:19 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T20:51:19.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:19 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:20.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T20:51:20.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T20:51:20.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[30184]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T20:51:20.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:20.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:20.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:20.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T20:51:20.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T20:51:20.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:20.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T20:51:20.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1056744707' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T20:51:20.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[42066]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T20:51:20.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:20.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:20.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:20.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T20:51:20.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T20:51:20.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:19 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:20.867 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.858+0000 7f61df44f700 1 -- 172.21.15.157:0/1954609041 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f61d8102920 msgr2=0x7f61d81091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:20.867 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.858+0000 7f61df44f700 1 --2- 172.21.15.157:0/1954609041 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f61d8102920 0x7f61d81091d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f61d000b0d0 tx=0x7f61d000b3e0 comp rx=0 tx=0).stop 2024-03-11T20:51:20.867 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.858+0000 7f61df44f700 1 -- 172.21.15.157:0/1954609041 shutdown_connections 2024-03-11T20:51:20.867 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.858+0000 7f61df44f700 1 --2- 172.21.15.157:0/1954609041 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f61d8102920 0x7f61d81091d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:20.867 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.858+0000 7f61df44f700 1 --2- 172.21.15.157:0/1954609041 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f61d8100d80 0x7f61d81011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:20.867 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.858+0000 7f61df44f700 1 --2- 172.21.15.157:0/1954609041 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61d8101f80 0x7f61d81023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:20.867 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.858+0000 7f61df44f700 1 -- 172.21.15.157:0/1954609041 >> 172.21.15.157:0/1954609041 conn(0x7f61d80fc340 msgr2=0x7f61d80fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:20.867 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.858+0000 7f61df44f700 1 -- 172.21.15.157:0/1954609041 shutdown_connections 2024-03-11T20:51:20.867 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.858+0000 7f61df44f700 1 -- 172.21.15.157:0/1954609041 wait complete. 2024-03-11T20:51:20.867 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.858+0000 7f61df44f700 1 Processor -- start 2024-03-11T20:51:20.867 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.858+0000 7f61df44f700 1 -- start start 2024-03-11T20:51:20.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.858+0000 7f61df44f700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61d8100d80 0x7f61d81975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:20.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.858+0000 7f61df44f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f61d8101f80 0x7f61d8197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:20.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61df44f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f61d8102920 0x7f61d819eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:20.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61df44f700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d810bdf0 con 0x7f61d8100d80 2024-03-11T20:51:20.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61df44f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d810bc70 con 0x7f61d8101f80 2024-03-11T20:51:20.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61df44f700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61d810bf70 con 0x7f61d8102920 2024-03-11T20:51:20.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61dc9ea700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f61d8101f80 0x7f61d8197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:20.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61dc9ea700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f61d8101f80 0x7f61d8197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:59488/0 (socket says 172.21.15.157:59488) 2024-03-11T20:51:20.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61dd1eb700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61d8100d80 0x7f61d81975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:20.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61dc9ea700 1 -- 172.21.15.157:0/1713296771 learned_addr learned my addr 172.21.15.157:0/1713296771 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:51:20.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61dd9ec700 1 --2- 172.21.15.157:0/1713296771 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f61d8102920 0x7f61d819eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:20.868 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61dc9ea700 1 -- 172.21.15.157:0/1713296771 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f61d8102920 msgr2=0x7f61d819eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:20.869 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61dc9ea700 1 --2- 172.21.15.157:0/1713296771 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f61d8102920 0x7f61d819eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:20.869 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61dc9ea700 1 -- 172.21.15.157:0/1713296771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61d8100d80 msgr2=0x7f61d81975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:20.869 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61dc9ea700 1 --2- 172.21.15.157:0/1713296771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61d8100d80 0x7f61d81975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:20.869 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61dc9ea700 1 -- 172.21.15.157:0/1713296771 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d819f1b0 con 0x7f61d8101f80 2024-03-11T20:51:20.869 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61dc9ea700 1 --2- 172.21.15.157:0/1713296771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f61d8101f80 0x7f61d8197ae0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f61c800c920 tx=0x7f61c800cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:20.869 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61ce7fc700 1 -- 172.21.15.157:0/1713296771 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c8004a30 con 0x7f61d8101f80 2024-03-11T20:51:20.869 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61ce7fc700 1 -- 172.21.15.157:0/1713296771 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f61c80054a0 con 0x7f61d8101f80 2024-03-11T20:51:20.869 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61df44f700 1 -- 172.21.15.157:0/1713296771 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d819f4a0 con 0x7f61d8101f80 2024-03-11T20:51:20.869 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61ce7fc700 1 -- 172.21.15.157:0/1713296771 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c8007ae0 con 0x7f61d8101f80 2024-03-11T20:51:20.869 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.859+0000 7f61df44f700 1 -- 172.21.15.157:0/1713296771 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d819f9e0 con 0x7f61d8101f80 2024-03-11T20:51:20.869 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.860+0000 7f61df44f700 1 -- 172.21.15.157:0/1713296771 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d8068aa0 con 0x7f61d8101f80 2024-03-11T20:51:20.870 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.861+0000 7f61ce7fc700 1 -- 172.21.15.157:0/1713296771 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 15) v1 ==== 91755+0+0 (secure 0 0 0) 0x7f61c8007cc0 con 0x7f61d8101f80 2024-03-11T20:51:20.870 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.861+0000 7f61ce7fc700 1 --2- 172.21.15.157:0/1713296771 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f61c406dd90 0x7f61c4070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:20.870 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.861+0000 7f61ce7fc700 1 -- 172.21.15.157:0/1713296771 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f61c808dab0 con 0x7f61d8101f80 2024-03-11T20:51:20.870 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.861+0000 7f61dd1eb700 1 --2- 172.21.15.157:0/1713296771 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f61c406dd90 0x7f61c4070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:20.870 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.862+0000 7f61dd1eb700 1 --2- 172.21.15.157:0/1713296771 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f61c406dd90 0x7f61c4070250 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f61d4005950 tx=0x7f61d400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:20.870 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:20.866+0000 7f61ce7fc700 1 -- 172.21.15.157:0/1713296771 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f61c8014030 con 0x7f61d8101f80 2024-03-11T20:51:21.033 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:20 smithi157 ceph-mon[33359]: Deploying daemon iscsi.iscsi.a on smithi157 2024-03-11T20:51:21.033 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:20 smithi157 ceph-mon[33359]: pgmap v203: 132 pgs: 103 active+clean, 3 creating+activating, 15 creating+peering, 11 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 1023 B/s wr, 15 op/s 2024-03-11T20:51:21.034 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:20 smithi157 ceph-mon[33359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T20:51:21.034 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:20 smithi157 ceph-mon[33359]: Cluster is now healthy 2024-03-11T20:51:21.034 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:20 smithi157 ceph-mon[33359]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T20:51:21.075 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:21.079+0000 7f61df44f700 1 -- 172.21.15.157:0/1713296771 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi157=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f61d8062220 con 0x7f61c406dd90 2024-03-11T20:51:21.085 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:21.089+0000 7f61ce7fc700 1 -- 172.21.15.157:0/1713296771 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f61d8062220 con 0x7f61c406dd90 2024-03-11T20:51:21.085 INFO:teuthology.orchestra.run.smithi157.stdout:Scheduled prometheus update... 2024-03-11T20:51:21.087 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:21.091+0000 7f61df44f700 1 -- 172.21.15.157:0/1713296771 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f61c406dd90 msgr2=0x7f61c4070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:21.087 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:21.091+0000 7f61df44f700 1 --2- 172.21.15.157:0/1713296771 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f61c406dd90 0x7f61c4070250 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f61d4005950 tx=0x7f61d400a300 comp rx=0 tx=0).stop 2024-03-11T20:51:21.087 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:21.091+0000 7f61df44f700 1 -- 172.21.15.157:0/1713296771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f61d8101f80 msgr2=0x7f61d8197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:21.087 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:21.091+0000 7f61df44f700 1 --2- 172.21.15.157:0/1713296771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f61d8101f80 0x7f61d8197ae0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f61c800c920 tx=0x7f61c800cc30 comp rx=0 tx=0).stop 2024-03-11T20:51:21.087 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:21.091+0000 7f61df44f700 1 -- 172.21.15.157:0/1713296771 shutdown_connections 2024-03-11T20:51:21.087 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:21.091+0000 7f61df44f700 1 --2- 172.21.15.157:0/1713296771 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f61c406dd90 0x7f61c4070250 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:21.088 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:21.091+0000 7f61df44f700 1 --2- 172.21.15.157:0/1713296771 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f61d8102920 0x7f61d819eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:21.088 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:21.091+0000 7f61df44f700 1 --2- 172.21.15.157:0/1713296771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61d8100d80 0x7f61d81975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:21.088 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:21.091+0000 7f61df44f700 1 --2- 172.21.15.157:0/1713296771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f61d8101f80 0x7f61d8197ae0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:21.088 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:21.091+0000 7f61df44f700 1 -- 172.21.15.157:0/1713296771 >> 172.21.15.157:0/1713296771 conn(0x7f61d80fc340 msgr2=0x7f61d806ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:21.088 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:21.091+0000 7f61df44f700 1 -- 172.21.15.157:0/1713296771 shutdown_connections 2024-03-11T20:51:21.088 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:21.091+0000 7f61df44f700 1 -- 172.21.15.157:0/1713296771 wait complete. 2024-03-11T20:51:21.129 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:20 smithi129 ceph-mon[30184]: Deploying daemon iscsi.iscsi.a on smithi157 2024-03-11T20:51:21.130 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:20 smithi129 ceph-mon[30184]: pgmap v203: 132 pgs: 103 active+clean, 3 creating+activating, 15 creating+peering, 11 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 1023 B/s wr, 15 op/s 2024-03-11T20:51:21.130 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:20 smithi129 ceph-mon[30184]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T20:51:21.130 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:20 smithi129 ceph-mon[30184]: Cluster is now healthy 2024-03-11T20:51:21.130 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:20 smithi129 ceph-mon[30184]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T20:51:21.130 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:20 smithi129 ceph-mon[42066]: Deploying daemon iscsi.iscsi.a on smithi157 2024-03-11T20:51:21.130 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:20 smithi129 ceph-mon[42066]: pgmap v203: 132 pgs: 103 active+clean, 3 creating+activating, 15 creating+peering, 11 unknown; 578 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 1023 B/s wr, 15 op/s 2024-03-11T20:51:21.130 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:20 smithi129 ceph-mon[42066]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T20:51:21.130 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:20 smithi129 ceph-mon[42066]: Cluster is now healthy 2024-03-11T20:51:21.130 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:20 smithi129 ceph-mon[42066]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T20:51:22.352 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:51:22 smithi157 systemd[1]: Starting Ceph iscsi.iscsi.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T20:51:22.352 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:22 smithi157 ceph-mon[33359]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi157=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:22.352 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:22 smithi157 ceph-mon[33359]: Saving service prometheus spec with placement smithi157=a;count:1 2024-03-11T20:51:22.352 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:22 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:22.352 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:22 smithi157 ceph-mon[33359]: pgmap v205: 132 pgs: 123 active+clean, 3 creating+activating, 6 creating+peering; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 2.2 KiB/s wr, 70 op/s 2024-03-11T20:51:22.377 DEBUG:teuthology.orchestra.run.smithi157:prometheus.a> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-11T20:51:22.379 INFO:tasks.cephadm:Adding node-exporter.a on smithi129 2024-03-11T20:51:22.379 INFO:tasks.cephadm:Adding node-exporter.b on smithi157 2024-03-11T20:51:22.379 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '2;smithi129=a;smithi157=b' 2024-03-11T20:51:22.403 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:22 smithi129 ceph-mon[30184]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi157=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:22.403 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:22 smithi129 ceph-mon[30184]: Saving service prometheus spec with placement smithi157=a;count:1 2024-03-11T20:51:22.403 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:22 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:22.403 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:22 smithi129 ceph-mon[30184]: pgmap v205: 132 pgs: 123 active+clean, 3 creating+activating, 6 creating+peering; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 2.2 KiB/s wr, 70 op/s 2024-03-11T20:51:22.403 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:22 smithi129 ceph-mon[42066]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi157=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:22.404 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:22 smithi129 ceph-mon[42066]: Saving service prometheus spec with placement smithi157=a;count:1 2024-03-11T20:51:22.404 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:22 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:22.404 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:22 smithi129 ceph-mon[42066]: pgmap v205: 132 pgs: 123 active+clean, 3 creating+activating, 6 creating+peering; 579 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 2.2 KiB/s wr, 70 op/s 2024-03-11T20:51:22.675 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:51:22 smithi157 podman[62335]: 2024-03-11 20:51:22.472839102 +0000 UTC m=+0.019872843 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T20:51:22.675 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:51:22 smithi157 podman[62335]: 2024-03-11 20:51:22.573675936 +0000 UTC m=+0.120709660 container create c3fc2ffaecb2d30341ac4b76753804ce9110a214b9672562743fb81c39345fec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-11T20:51:22.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:51:22 smithi157 podman[62335]: 2024-03-11 20:51:22.842915879 +0000 UTC m=+0.389949602 container init c3fc2ffaecb2d30341ac4b76753804ce9110a214b9672562743fb81c39345fec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T20:51:22.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:51:22 smithi157 podman[62335]: 2024-03-11 20:51:22.848811532 +0000 UTC m=+0.395845257 container start c3fc2ffaecb2d30341ac4b76753804ce9110a214b9672562743fb81c39345fec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T20:51:23.016 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:51:23.235 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:51:22 smithi157 bash[62335]: c3fc2ffaecb2d30341ac4b76753804ce9110a214b9672562743fb81c39345fec 2024-03-11T20:51:23.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:51:22 smithi157 systemd[1]: Started Ceph iscsi.iscsi.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T20:51:23.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:51:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug Started the configuration object watcher 2024-03-11T20:51:23.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:51:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug Checking for config object changes every 1s 2024-03-11T20:51:23.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:51:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug Processing osd blocklist entries for this node 2024-03-11T20:51:23.509 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:51:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug Reading the configuration object to update local LIO configuration 2024-03-11T20:51:23.509 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:51:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug Configuration does not have an entry for this host(smithi157.front.sepia.ceph.com) - nothing to define to LIO 2024-03-11T20:51:23.509 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:51:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-11T20:51:24.053 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:24 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:24.053 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:24 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:24.054 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:24 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:24.054 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:24 smithi157 ceph-mon[33359]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T20:51:24.054 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:24 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:24.054 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:24 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:51:24.054 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.157:0/4017010394' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T20:51:24.054 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:24 smithi157 ceph-mon[33359]: pgmap v206: 132 pgs: 132 active+clean; 580 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 3.3 KiB/s wr, 106 op/s 2024-03-11T20:51:24.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:24 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:24.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:24 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:24.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:24 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:24 smithi129 ceph-mon[30184]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T20:51:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:24 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:24 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:51:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.157:0/4017010394' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T20:51:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:24 smithi129 ceph-mon[30184]: pgmap v206: 132 pgs: 132 active+clean; 580 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 3.3 KiB/s wr, 106 op/s 2024-03-11T20:51:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:24 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:24 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:24 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:24 smithi129 ceph-mon[42066]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T20:51:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:24 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:24 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:51:24.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.157:0/4017010394' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T20:51:24.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:24 smithi129 ceph-mon[42066]: pgmap v206: 132 pgs: 132 active+clean; 580 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 3.3 KiB/s wr, 106 op/s 2024-03-11T20:51:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.130+0000 7f90a2d0c700 1 -- 172.21.15.157:0/799969213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f909c1000d0 msgr2=0x7f909c1004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.130+0000 7f90a2d0c700 1 --2- 172.21.15.157:0/799969213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f909c1000d0 0x7f909c1004f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9090009990 tx=0x7f9090009ca0 comp rx=0 tx=0).stop 2024-03-11T20:51:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.130+0000 7f90a2d0c700 1 -- 172.21.15.157:0/799969213 shutdown_connections 2024-03-11T20:51:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.130+0000 7f90a2d0c700 1 --2- 172.21.15.157:0/799969213 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f909c101800 0x7f909c1060a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.130+0000 7f90a2d0c700 1 --2- 172.21.15.157:0/799969213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f909c1000d0 0x7f909c1004f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.130+0000 7f90a2d0c700 1 --2- 172.21.15.157:0/799969213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f909c100e40 0x7f909c1012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.130+0000 7f90a2d0c700 1 -- 172.21.15.157:0/799969213 >> 172.21.15.157:0/799969213 conn(0x7f909c0fba50 msgr2=0x7f909c0fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.130+0000 7f90a2d0c700 1 -- 172.21.15.157:0/799969213 shutdown_connections 2024-03-11T20:51:25.128 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.130+0000 7f90a2d0c700 1 -- 172.21.15.157:0/799969213 wait complete. 2024-03-11T20:51:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a2d0c700 1 Processor -- start 2024-03-11T20:51:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a2d0c700 1 -- start start 2024-03-11T20:51:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a2d0c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f909c1000d0 0x7f909c19db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a2d0c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f909c100e40 0x7f909c19e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a2d0c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f909c101800 0x7f909c1a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a2d0c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c1134a0 con 0x7f909c100e40 2024-03-11T20:51:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a2d0c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c113320 con 0x7f909c1000d0 2024-03-11T20:51:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a2d0c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909c113620 con 0x7f909c101800 2024-03-11T20:51:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a0aa8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f909c1000d0 0x7f909c19db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a0aa8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f909c1000d0 0x7f909c19db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:59564/0 (socket says 172.21.15.157:59564) 2024-03-11T20:51:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a0aa8700 1 -- 172.21.15.157:0/2913056541 learned_addr learned my addr 172.21.15.157:0/2913056541 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:51:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a12a9700 1 --2- 172.21.15.157:0/2913056541 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f909c101800 0x7f909c1a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f909bfff700 1 --2- 172.21.15.157:0/2913056541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f909c100e40 0x7f909c19e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:25.129 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a0aa8700 1 -- 172.21.15.157:0/2913056541 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f909c101800 msgr2=0x7f909c1a5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a0aa8700 1 --2- 172.21.15.157:0/2913056541 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f909c101800 0x7f909c1a5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a0aa8700 1 -- 172.21.15.157:0/2913056541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f909c100e40 msgr2=0x7f909c19e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a0aa8700 1 --2- 172.21.15.157:0/2913056541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f909c100e40 0x7f909c19e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a0aa8700 1 -- 172.21.15.157:0/2913056541 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c1a57e0 con 0x7f909c1000d0 2024-03-11T20:51:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f90a0aa8700 1 --2- 172.21.15.157:0/2913056541 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f909c1000d0 0x7f909c19db80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f90900098e0 tx=0x7f909000f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.131+0000 7f9099ffb700 1 -- 172.21.15.157:0/2913056541 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090027070 con 0x7f909c1000d0 2024-03-11T20:51:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.132+0000 7f90a2d0c700 1 -- 172.21.15.157:0/2913056541 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909c1a5a70 con 0x7f909c1000d0 2024-03-11T20:51:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.132+0000 7f9099ffb700 1 -- 172.21.15.157:0/2913056541 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9090022470 con 0x7f909c1000d0 2024-03-11T20:51:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.132+0000 7f9099ffb700 1 -- 172.21.15.157:0/2913056541 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090003a90 con 0x7f909c1000d0 2024-03-11T20:51:25.130 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.132+0000 7f90a2d0c700 1 -- 172.21.15.157:0/2913056541 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909c1a5ec0 con 0x7f909c1000d0 2024-03-11T20:51:25.131 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.133+0000 7f90a2d0c700 1 -- 172.21.15.157:0/2913056541 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f909c068aa0 con 0x7f909c1000d0 2024-03-11T20:51:25.131 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.133+0000 7f9099ffb700 1 -- 172.21.15.157:0/2913056541 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 16) v1 ==== 91806+0+0 (secure 0 0 0) 0x7f909002a070 con 0x7f909c1000d0 2024-03-11T20:51:25.131 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.134+0000 7f9099ffb700 1 --2- 172.21.15.157:0/2913056541 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f908c06de20 0x7f908c0702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:25.131 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.134+0000 7f9099ffb700 1 -- 172.21.15.157:0/2913056541 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f909008f190 con 0x7f909c1000d0 2024-03-11T20:51:25.131 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.134+0000 7f909bfff700 1 --2- 172.21.15.157:0/2913056541 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f908c06de20 0x7f908c0702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:25.132 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.134+0000 7f909bfff700 1 --2- 172.21.15.157:0/2913056541 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f908c06de20 0x7f908c0702e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9088005fd0 tx=0x7f908800a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:25.134 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.138+0000 7f9099ffb700 1 -- 172.21.15.157:0/2913056541 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f9090013070 con 0x7f909c1000d0 2024-03-11T20:51:25.134 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:25 smithi157 ceph-mon[33359]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-11T20:51:25.135 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:25 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:25.344 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.348+0000 7f90a2d0c700 1 -- 172.21.15.157:0/2913056541 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi129=a;smithi157=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f909c062220 con 0x7f908c06de20 2024-03-11T20:51:25.352 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.356+0000 7f9099ffb700 1 -- 172.21.15.157:0/2913056541 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f909c062220 con 0x7f908c06de20 2024-03-11T20:51:25.352 INFO:teuthology.orchestra.run.smithi157.stdout:Scheduled node-exporter update... 2024-03-11T20:51:25.354 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.358+0000 7f90a2d0c700 1 -- 172.21.15.157:0/2913056541 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f908c06de20 msgr2=0x7f908c0702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:25.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.358+0000 7f90a2d0c700 1 --2- 172.21.15.157:0/2913056541 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f908c06de20 0x7f908c0702e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9088005fd0 tx=0x7f908800a560 comp rx=0 tx=0).stop 2024-03-11T20:51:25.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.358+0000 7f90a2d0c700 1 -- 172.21.15.157:0/2913056541 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f909c1000d0 msgr2=0x7f909c19db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:25.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.358+0000 7f90a2d0c700 1 --2- 172.21.15.157:0/2913056541 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f909c1000d0 0x7f909c19db80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f90900098e0 tx=0x7f909000f740 comp rx=0 tx=0).stop 2024-03-11T20:51:25.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.358+0000 7f90a2d0c700 1 -- 172.21.15.157:0/2913056541 shutdown_connections 2024-03-11T20:51:25.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.358+0000 7f90a2d0c700 1 --2- 172.21.15.157:0/2913056541 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f908c06de20 0x7f908c0702e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:25.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.358+0000 7f90a2d0c700 1 --2- 172.21.15.157:0/2913056541 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f909c101800 0x7f909c1a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:25.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.358+0000 7f90a2d0c700 1 --2- 172.21.15.157:0/2913056541 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f909c1000d0 0x7f909c19db80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:25.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.358+0000 7f90a2d0c700 1 --2- 172.21.15.157:0/2913056541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f909c100e40 0x7f909c19e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:25.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.358+0000 7f90a2d0c700 1 -- 172.21.15.157:0/2913056541 >> 172.21.15.157:0/2913056541 conn(0x7f909c0fba50 msgr2=0x7f909c102080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:25.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.359+0000 7f90a2d0c700 1 -- 172.21.15.157:0/2913056541 shutdown_connections 2024-03-11T20:51:25.355 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:25.359+0000 7f90a2d0c700 1 -- 172.21.15.157:0/2913056541 wait complete. 2024-03-11T20:51:25.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:25 smithi129 ceph-mon[30184]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-11T20:51:25.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:25 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:25.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:25 smithi129 ceph-mon[42066]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-11T20:51:25.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:25 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:26.218 DEBUG:teuthology.orchestra.run.smithi129:node-exporter.a> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-11T20:51:26.221 DEBUG:teuthology.orchestra.run.smithi157:node-exporter.b> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-11T20:51:26.223 INFO:tasks.cephadm:Adding alertmanager.a on smithi129 2024-03-11T20:51:26.223 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi129=a' 2024-03-11T20:51:26.468 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:26 smithi157 ceph-mon[33359]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi129=a;smithi157=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:26.468 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:26 smithi157 ceph-mon[33359]: Saving service node-exporter spec with placement smithi129=a;smithi157=b;count:2 2024-03-11T20:51:26.468 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:26 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:26.468 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:26 smithi157 ceph-mon[33359]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.4 KiB/s wr, 142 op/s 2024-03-11T20:51:26.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:26 smithi129 ceph-mon[30184]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi129=a;smithi157=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:26.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:26 smithi129 ceph-mon[30184]: Saving service node-exporter spec with placement smithi129=a;smithi157=b;count:2 2024-03-11T20:51:26.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:26 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:26.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:26 smithi129 ceph-mon[30184]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.4 KiB/s wr, 142 op/s 2024-03-11T20:51:26.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:26 smithi129 ceph-mon[42066]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi129=a;smithi157=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:26.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:26 smithi129 ceph-mon[42066]: Saving service node-exporter spec with placement smithi129=a;smithi157=b;count:2 2024-03-11T20:51:26.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:26 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:26.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:26 smithi129 ceph-mon[42066]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.4 KiB/s wr, 142 op/s 2024-03-11T20:51:26.848 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:51:28.503 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.506+0000 7f4b6c9a0700 1 -- 172.21.15.157:0/2164303256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b68103450 msgr2=0x7f4b68103870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:28.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.506+0000 7f4b6c9a0700 1 --2- 172.21.15.157:0/2164303256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b68103450 0x7f4b68103870 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4b500099c0 tx=0x7f4b50009cd0 comp rx=0 tx=0).stop 2024-03-11T20:51:28.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.506+0000 7f4b6c9a0700 1 -- 172.21.15.157:0/2164303256 shutdown_connections 2024-03-11T20:51:28.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.506+0000 7f4b6c9a0700 1 --2- 172.21.15.157:0/2164303256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b68100890 0x7f4b6810a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:28.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.506+0000 7f4b6c9a0700 1 --2- 172.21.15.157:0/2164303256 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b68103450 0x7f4b68103870 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:28.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.506+0000 7f4b6c9a0700 1 --2- 172.21.15.157:0/2164303256 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4b680ffed0 0x7f4b68100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:28.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.506+0000 7f4b6c9a0700 1 -- 172.21.15.157:0/2164303256 >> 172.21.15.157:0/2164303256 conn(0x7f4b680fba70 msgr2=0x7f4b680fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:28.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.506+0000 7f4b6c9a0700 1 -- 172.21.15.157:0/2164303256 shutdown_connections 2024-03-11T20:51:28.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.506+0000 7f4b6c9a0700 1 -- 172.21.15.157:0/2164303256 wait complete. 2024-03-11T20:51:28.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.507+0000 7f4b6c9a0700 1 Processor -- start 2024-03-11T20:51:28.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.507+0000 7f4b6c9a0700 1 -- start start 2024-03-11T20:51:28.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.507+0000 7f4b6c9a0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b680ffed0 0x7f4b681976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:28.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.507+0000 7f4b6c9a0700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b68100890 0x7f4b68197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:28.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.507+0000 7f4b6c9a0700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4b68103450 0x7f4b6819ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:28.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.507+0000 7f4b6c9a0700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6810cfc0 con 0x7f4b68100890 2024-03-11T20:51:28.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.507+0000 7f4b6c9a0700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6810ce40 con 0x7f4b680ffed0 2024-03-11T20:51:28.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.507+0000 7f4b6c9a0700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6810d140 con 0x7f4b68103450 2024-03-11T20:51:28.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.507+0000 7f4b6659c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b680ffed0 0x7f4b681976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:28.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.507+0000 7f4b6659c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b680ffed0 0x7f4b681976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:59568/0 (socket says 172.21.15.157:59568) 2024-03-11T20:51:28.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.507+0000 7f4b6659c700 1 -- 172.21.15.157:0/3363880324 learned_addr learned my addr 172.21.15.157:0/3363880324 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:51:28.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.508+0000 7f4b65d9b700 1 --2- 172.21.15.157:0/3363880324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b68100890 0x7f4b68197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:28.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.508+0000 7f4b66d9d700 1 --2- 172.21.15.157:0/3363880324 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4b68103450 0x7f4b6819ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:28.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.508+0000 7f4b6659c700 1 -- 172.21.15.157:0/3363880324 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4b68103450 msgr2=0x7f4b6819ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:28.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.508+0000 7f4b6659c700 1 --2- 172.21.15.157:0/3363880324 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4b68103450 0x7f4b6819ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:28.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.508+0000 7f4b6659c700 1 -- 172.21.15.157:0/3363880324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b68100890 msgr2=0x7f4b68197c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:28.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.508+0000 7f4b6659c700 1 --2- 172.21.15.157:0/3363880324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b68100890 0x7f4b68197c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:28.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.508+0000 7f4b6659c700 1 -- 172.21.15.157:0/3363880324 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6819f350 con 0x7f4b680ffed0 2024-03-11T20:51:28.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.508+0000 7f4b65d9b700 1 --2- 172.21.15.157:0/3363880324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b68100890 0x7f4b68197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T20:51:28.506 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.508+0000 7f4b6659c700 1 --2- 172.21.15.157:0/3363880324 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b680ffed0 0x7f4b681976f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4b500047b0 tx=0x7f4b5000bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:28.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.508+0000 7f4b5f7fe700 1 -- 172.21.15.157:0/3363880324 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b50027070 con 0x7f4b680ffed0 2024-03-11T20:51:28.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.508+0000 7f4b6c9a0700 1 -- 172.21.15.157:0/3363880324 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b6819f5e0 con 0x7f4b680ffed0 2024-03-11T20:51:28.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.508+0000 7f4b5f7fe700 1 -- 172.21.15.157:0/3363880324 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4b50022470 con 0x7f4b680ffed0 2024-03-11T20:51:28.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.508+0000 7f4b5f7fe700 1 -- 172.21.15.157:0/3363880324 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5000f6c0 con 0x7f4b680ffed0 2024-03-11T20:51:28.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.508+0000 7f4b6c9a0700 1 -- 172.21.15.157:0/3363880324 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b6819fb10 con 0x7f4b680ffed0 2024-03-11T20:51:28.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.509+0000 7f4b6c9a0700 1 -- 172.21.15.157:0/3363880324 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b68068aa0 con 0x7f4b680ffed0 2024-03-11T20:51:28.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.510+0000 7f4b5f7fe700 1 -- 172.21.15.157:0/3363880324 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 16) v1 ==== 91806+0+0 (secure 0 0 0) 0x7f4b5002a070 con 0x7f4b680ffed0 2024-03-11T20:51:28.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.510+0000 7f4b5f7fe700 1 --2- 172.21.15.157:0/3363880324 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4b5406de20 0x7f4b540702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:28.507 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.510+0000 7f4b5f7fe700 1 -- 172.21.15.157:0/3363880324 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7148+0+0 (secure 0 0 0) 0x7f4b5008f1b0 con 0x7f4b680ffed0 2024-03-11T20:51:28.508 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.510+0000 7f4b65d9b700 1 --2- 172.21.15.157:0/3363880324 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4b5406de20 0x7f4b540702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:28.508 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.511+0000 7f4b65d9b700 1 --2- 172.21.15.157:0/3363880324 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4b5406de20 0x7f4b540702e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4b6806a490 tx=0x7f4b58008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:28.511 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.515+0000 7f4b5f7fe700 1 -- 172.21.15.157:0/3363880324 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f4b50013070 con 0x7f4b680ffed0 2024-03-11T20:51:28.726 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.730+0000 7f4b6c9a0700 1 -- 172.21.15.157:0/3363880324 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b6802d970 con 0x7f4b5406de20 2024-03-11T20:51:28.735 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.739+0000 7f4b5f7fe700 1 -- 172.21.15.157:0/3363880324 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f4b6802d970 con 0x7f4b5406de20 2024-03-11T20:51:28.735 INFO:teuthology.orchestra.run.smithi157.stdout:Scheduled alertmanager update... 2024-03-11T20:51:28.737 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.741+0000 7f4b6c9a0700 1 -- 172.21.15.157:0/3363880324 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4b5406de20 msgr2=0x7f4b540702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:28.737 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.741+0000 7f4b6c9a0700 1 --2- 172.21.15.157:0/3363880324 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4b5406de20 0x7f4b540702e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4b6806a490 tx=0x7f4b58008040 comp rx=0 tx=0).stop 2024-03-11T20:51:28.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.741+0000 7f4b6c9a0700 1 -- 172.21.15.157:0/3363880324 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b680ffed0 msgr2=0x7f4b681976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:28.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.741+0000 7f4b6c9a0700 1 --2- 172.21.15.157:0/3363880324 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b680ffed0 0x7f4b681976f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4b500047b0 tx=0x7f4b5000bfd0 comp rx=0 tx=0).stop 2024-03-11T20:51:28.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.741+0000 7f4b6c9a0700 1 -- 172.21.15.157:0/3363880324 shutdown_connections 2024-03-11T20:51:28.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.741+0000 7f4b6c9a0700 1 --2- 172.21.15.157:0/3363880324 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f4b5406de20 0x7f4b540702e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:28.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.741+0000 7f4b6c9a0700 1 --2- 172.21.15.157:0/3363880324 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4b68103450 0x7f4b6819ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:28.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.741+0000 7f4b6c9a0700 1 --2- 172.21.15.157:0/3363880324 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b680ffed0 0x7f4b681976f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:28.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.741+0000 7f4b6c9a0700 1 --2- 172.21.15.157:0/3363880324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b68100890 0x7f4b68197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:28.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.741+0000 7f4b6c9a0700 1 -- 172.21.15.157:0/3363880324 >> 172.21.15.157:0/3363880324 conn(0x7f4b680fba70 msgr2=0x7f4b680fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:28.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.742+0000 7f4b6c9a0700 1 -- 172.21.15.157:0/3363880324 shutdown_connections 2024-03-11T20:51:28.738 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:28.742+0000 7f4b6c9a0700 1 -- 172.21.15.157:0/3363880324 wait complete. 2024-03-11T20:51:28.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:28 smithi129 ceph-mon[30184]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.9 KiB/s wr, 127 op/s 2024-03-11T20:51:28.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:28 smithi129 ceph-mon[42066]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.9 KiB/s wr, 127 op/s 2024-03-11T20:51:28.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:28 smithi157 ceph-mon[33359]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.9 KiB/s wr, 127 op/s 2024-03-11T20:51:29.819 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:29 smithi157 ceph-mon[33359]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:29.819 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:29 smithi157 ceph-mon[33359]: Saving service alertmanager spec with placement smithi129=a;count:1 2024-03-11T20:51:29.819 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:29.819 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:29.819 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:29.819 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:29.819 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:51:29.819 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:29.819 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-11T20:51:29.820 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-11T20:51:29.820 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-11T20:51:29.820 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:29 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:29.856 DEBUG:teuthology.orchestra.run.smithi129:alertmanager.a> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-11T20:51:29.859 INFO:tasks.cephadm:Adding grafana.a on smithi157 2024-03-11T20:51:29.859 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi157=a' 2024-03-11T20:51:30.109 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:51:30.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[30184]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:30.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[30184]: Saving service alertmanager spec with placement smithi129=a;count:1 2024-03-11T20:51:30.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:30.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:30.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:30.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:30.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:51:30.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:30.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-11T20:51:30.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-11T20:51:30.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-11T20:51:30.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:30.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[42066]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:30.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[42066]: Saving service alertmanager spec with placement smithi129=a;count:1 2024-03-11T20:51:30.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:30.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:30.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:30.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:51:30.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:51:30.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:30.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-11T20:51:30.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-11T20:51:30.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-11T20:51:30.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:29 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:30 smithi157 ceph-mon[33359]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 3.4 KiB/s wr, 108 op/s 2024-03-11T20:51:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:30 smithi157 ceph-mon[33359]: Deploying daemon prometheus.a on smithi157 2024-03-11T20:51:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:30 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-11T20:51:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:30 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-11T20:51:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:30 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-11T20:51:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:30 smithi157 ceph-mon[33359]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T20:51:31.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:30 smithi129 ceph-mon[30184]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 3.4 KiB/s wr, 108 op/s 2024-03-11T20:51:31.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:30 smithi129 ceph-mon[30184]: Deploying daemon prometheus.a on smithi157 2024-03-11T20:51:31.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:30 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-11T20:51:31.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:30 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-11T20:51:31.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:30 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-11T20:51:31.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:30 smithi129 ceph-mon[30184]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T20:51:31.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:30 smithi129 ceph-mon[42066]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 3.4 KiB/s wr, 108 op/s 2024-03-11T20:51:31.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:30 smithi129 ceph-mon[42066]: Deploying daemon prometheus.a on smithi157 2024-03-11T20:51:31.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:30 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-11T20:51:31.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:30 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-11T20:51:31.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:30 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-11T20:51:31.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:30 smithi129 ceph-mon[42066]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T20:51:31.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.795+0000 7f885860a700 1 -- 172.21.15.157:0/1610634911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8850100120 msgr2=0x7f8850100540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:31.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.795+0000 7f885860a700 1 --2- 172.21.15.157:0/1610634911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8850100120 0x7f8850100540 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f884c009990 tx=0x7f884c009ca0 comp rx=0 tx=0).stop 2024-03-11T20:51:31.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.795+0000 7f885860a700 1 -- 172.21.15.157:0/1610634911 shutdown_connections 2024-03-11T20:51:31.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.795+0000 7f885860a700 1 --2- 172.21.15.157:0/1610634911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8850103aa0 0x7f885010a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:31.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.795+0000 7f885860a700 1 --2- 172.21.15.157:0/1610634911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8850100120 0x7f8850100540 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:31.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.795+0000 7f885860a700 1 --2- 172.21.15.157:0/1610634911 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8850103100 0x7f8850103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:31.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.795+0000 7f885860a700 1 -- 172.21.15.157:0/1610634911 >> 172.21.15.157:0/1610634911 conn(0x7f88500fba80 msgr2=0x7f88500fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:31.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.795+0000 7f885860a700 1 -- 172.21.15.157:0/1610634911 shutdown_connections 2024-03-11T20:51:31.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.795+0000 7f885860a700 1 -- 172.21.15.157:0/1610634911 wait complete. 2024-03-11T20:51:31.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.795+0000 7f885860a700 1 Processor -- start 2024-03-11T20:51:31.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.795+0000 7f885860a700 1 -- start start 2024-03-11T20:51:31.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f885860a700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8850100120 0x7f8850197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:31.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f885860a700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8850103100 0x7f8850197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:31.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f885860a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8850103aa0 0x7f885019eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:31.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f885860a700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885010ce90 con 0x7f8850103100 2024-03-11T20:51:31.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f885860a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885010cd10 con 0x7f8850103aa0 2024-03-11T20:51:31.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f885860a700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885010d010 con 0x7f8850100120 2024-03-11T20:51:31.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f8856ba7700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8850103aa0 0x7f885019eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:31.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f8856ba7700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8850103aa0 0x7f885019eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:42724/0 (socket says 172.21.15.157:42724) 2024-03-11T20:51:31.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f8856ba7700 1 -- 172.21.15.157:0/2762199042 learned_addr learned my addr 172.21.15.157:0/2762199042 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:51:31.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f8855ba5700 1 --2- 172.21.15.157:0/2762199042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8850103100 0x7f8850197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:31.797 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f88563a6700 1 --2- 172.21.15.157:0/2762199042 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8850100120 0x7f8850197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:31.798 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f8856ba7700 1 -- 172.21.15.157:0/2762199042 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8850100120 msgr2=0x7f8850197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:31.798 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f8856ba7700 1 --2- 172.21.15.157:0/2762199042 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8850100120 0x7f8850197570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:31.798 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f8856ba7700 1 -- 172.21.15.157:0/2762199042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8850103100 msgr2=0x7f8850197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:31.798 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f8856ba7700 1 --2- 172.21.15.157:0/2762199042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8850103100 0x7f8850197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:31.798 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f8856ba7700 1 -- 172.21.15.157:0/2762199042 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885019f1d0 con 0x7f8850103aa0 2024-03-11T20:51:31.798 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f8856ba7700 1 --2- 172.21.15.157:0/2762199042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8850103aa0 0x7f885019eb30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8848007ee0 tx=0x7f884800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:31.798 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f88477fe700 1 -- 172.21.15.157:0/2762199042 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8848004bb0 con 0x7f8850103aa0 2024-03-11T20:51:31.798 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f88477fe700 1 -- 172.21.15.157:0/2762199042 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8848004d50 con 0x7f8850103aa0 2024-03-11T20:51:31.798 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f885860a700 1 -- 172.21.15.157:0/2762199042 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f885019f4c0 con 0x7f8850103aa0 2024-03-11T20:51:31.798 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f88477fe700 1 -- 172.21.15.157:0/2762199042 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88480057d0 con 0x7f8850103aa0 2024-03-11T20:51:31.798 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.796+0000 7f885860a700 1 -- 172.21.15.157:0/2762199042 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f885019f9b0 con 0x7f8850103aa0 2024-03-11T20:51:31.798 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.797+0000 7f885860a700 1 -- 172.21.15.157:0/2762199042 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8850068aa0 con 0x7f8850103aa0 2024-03-11T20:51:31.798 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.798+0000 7f88477fe700 1 -- 172.21.15.157:0/2762199042 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 16) v1 ==== 91806+0+0 (secure 0 0 0) 0x7f884801a020 con 0x7f8850103aa0 2024-03-11T20:51:31.799 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.798+0000 7f88477fe700 1 --2- 172.21.15.157:0/2762199042 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f883c06de20 0x7f883c0702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:31.799 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.798+0000 7f88477fe700 1 -- 172.21.15.157:0/2762199042 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7207+0+0 (secure 0 0 0) 0x7f884808d700 con 0x7f8850103aa0 2024-03-11T20:51:31.799 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.798+0000 7f88563a6700 1 --2- 172.21.15.157:0/2762199042 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f883c06de20 0x7f883c0702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:31.799 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.799+0000 7f88563a6700 1 --2- 172.21.15.157:0/2762199042 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f883c06de20 0x7f883c0702e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f884c0098e0 tx=0x7f884c0036f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:31.800 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:31.803+0000 7f88477fe700 1 -- 172.21.15.157:0/2762199042 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f8848010070 con 0x7f8850103aa0 2024-03-11T20:51:32.016 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:32.020+0000 7f885860a700 1 -- 172.21.15.157:0/2762199042 --> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi157=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f8850062220 con 0x7f883c06de20 2024-03-11T20:51:32.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:32.027+0000 7f88477fe700 1 -- 172.21.15.157:0/2762199042 <== mgr.14152 v2:172.21.15.129:6800/2866184010 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f8850062220 con 0x7f883c06de20 2024-03-11T20:51:32.024 INFO:teuthology.orchestra.run.smithi157.stdout:Scheduled grafana update... 2024-03-11T20:51:32.026 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:32.029+0000 7f885860a700 1 -- 172.21.15.157:0/2762199042 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f883c06de20 msgr2=0x7f883c0702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:32.026 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:32.029+0000 7f885860a700 1 --2- 172.21.15.157:0/2762199042 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f883c06de20 0x7f883c0702e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f884c0098e0 tx=0x7f884c0036f0 comp rx=0 tx=0).stop 2024-03-11T20:51:32.026 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:32.029+0000 7f885860a700 1 -- 172.21.15.157:0/2762199042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8850103aa0 msgr2=0x7f885019eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:32.026 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:32.029+0000 7f885860a700 1 --2- 172.21.15.157:0/2762199042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8850103aa0 0x7f885019eb30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8848007ee0 tx=0x7f884800d3b0 comp rx=0 tx=0).stop 2024-03-11T20:51:32.026 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:32.029+0000 7f885860a700 1 -- 172.21.15.157:0/2762199042 shutdown_connections 2024-03-11T20:51:32.026 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:32.029+0000 7f885860a700 1 --2- 172.21.15.157:0/2762199042 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f883c06de20 0x7f883c0702e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:32.027 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:32.029+0000 7f885860a700 1 --2- 172.21.15.157:0/2762199042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8850103aa0 0x7f885019eb30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:32.027 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:32.029+0000 7f885860a700 1 --2- 172.21.15.157:0/2762199042 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8850100120 0x7f8850197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:32.027 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:32.029+0000 7f885860a700 1 --2- 172.21.15.157:0/2762199042 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8850103100 0x7f8850197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:32.027 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:32.029+0000 7f885860a700 1 -- 172.21.15.157:0/2762199042 >> 172.21.15.157:0/2762199042 conn(0x7f88500fba80 msgr2=0x7f88500fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:32.027 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:32.029+0000 7f885860a700 1 -- 172.21.15.157:0/2762199042 shutdown_connections 2024-03-11T20:51:32.027 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:32.029+0000 7f885860a700 1 -- 172.21.15.157:0/2762199042 wait complete. 2024-03-11T20:51:32.915 DEBUG:teuthology.orchestra.run.smithi157:grafana.a> sudo journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@grafana.a.service 2024-03-11T20:51:32.918 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T20:51:32.918 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T20:51:33.147 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:32 smithi157 ceph-mon[33359]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.6 KiB/s wr, 79 op/s 2024-03-11T20:51:33.147 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:32 smithi157 ceph-mon[33359]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T20:51:33.147 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:32 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:33.157 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:51:33.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:32 smithi129 ceph-mon[30184]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.6 KiB/s wr, 79 op/s 2024-03-11T20:51:33.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:32 smithi129 ceph-mon[30184]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T20:51:33.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:32 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:33.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:32 smithi129 ceph-mon[42066]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 2.6 KiB/s wr, 79 op/s 2024-03-11T20:51:33.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:32 smithi129 ceph-mon[42066]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T20:51:33.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:32 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:33.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:51:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:51:33.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:33 smithi129 ceph-mon[42066]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi157=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:33.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:33 smithi129 ceph-mon[42066]: Saving service grafana spec with placement smithi157=a;count:1 2024-03-11T20:51:33.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:33 smithi129 ceph-mon[30184]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi157=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:33.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:33 smithi129 ceph-mon[30184]: Saving service grafana spec with placement smithi157=a;count:1 2024-03-11T20:51:34.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:33 smithi157 ceph-mon[33359]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi157=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T20:51:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:33 smithi157 ceph-mon[33359]: Saving service grafana spec with placement smithi157=a;count:1 2024-03-11T20:51:34.886 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:34 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:34.887 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:34 smithi129 ceph-mon[30184]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s; 13 B/s, 0 objects/s recovering 2024-03-11T20:51:34.887 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:34 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:34.887 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:34 smithi129 ceph-mon[42066]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s; 13 B/s, 0 objects/s recovering 2024-03-11T20:51:34.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 -- 172.21.15.129:0/2902953353 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47ec103970 msgr2=0x7f47ec10a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:34.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 --2- 172.21.15.129:0/2902953353 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47ec103970 0x7f47ec10a290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f47e400b0a0 tx=0x7f47e400b3b0 comp rx=0 tx=0).stop 2024-03-11T20:51:34.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 -- 172.21.15.129:0/2902953353 shutdown_connections 2024-03-11T20:51:34.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 --2- 172.21.15.129:0/2902953353 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47ec103970 0x7f47ec10a290 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:34.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 --2- 172.21.15.129:0/2902953353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47ec101e50 0x7f47ec102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:34.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 --2- 172.21.15.129:0/2902953353 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47ec102fb0 0x7f47ec103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:34.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 -- 172.21.15.129:0/2902953353 >> 172.21.15.129:0/2902953353 conn(0x7f47ec0fd3f0 msgr2=0x7f47ec0ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:34.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 -- 172.21.15.129:0/2902953353 shutdown_connections 2024-03-11T20:51:34.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 -- 172.21.15.129:0/2902953353 wait complete. 2024-03-11T20:51:34.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 Processor -- start 2024-03-11T20:51:34.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 -- start start 2024-03-11T20:51:34.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47ec101e50 0x7f47ec075b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:34.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47ec102fb0 0x7f47ec0741d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:34.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47ec103970 0x7f47ec074710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:34.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec10cde0 con 0x7f47ec103970 2024-03-11T20:51:34.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec10cc60 con 0x7f47ec102fb0 2024-03-11T20:51:34.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.908+0000 7f47f2ae8700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47ec10cf60 con 0x7f47ec101e50 2024-03-11T20:51:34.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.909+0000 7f47f0884700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47ec101e50 0x7f47ec075b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:34.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.909+0000 7f47ebfff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47ec102fb0 0x7f47ec0741d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:34.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.909+0000 7f47f1085700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47ec103970 0x7f47ec074710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:34.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.909+0000 7f47f0884700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47ec101e50 0x7f47ec075b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:57538/0 (socket says 172.21.15.129:57538) 2024-03-11T20:51:34.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.909+0000 7f47f0884700 1 -- 172.21.15.129:0/2837856214 learned_addr learned my addr 172.21.15.129:0/2837856214 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:51:34.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.909+0000 7f47f1085700 1 -- 172.21.15.129:0/2837856214 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47ec101e50 msgr2=0x7f47ec075b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:34.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.909+0000 7f47f1085700 1 --2- 172.21.15.129:0/2837856214 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47ec101e50 0x7f47ec075b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:34.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.909+0000 7f47f1085700 1 -- 172.21.15.129:0/2837856214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47ec102fb0 msgr2=0x7f47ec0741d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:34.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.909+0000 7f47f1085700 1 --2- 172.21.15.129:0/2837856214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47ec102fb0 0x7f47ec0741d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:34.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.909+0000 7f47f1085700 1 -- 172.21.15.129:0/2837856214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47ec074fc0 con 0x7f47ec103970 2024-03-11T20:51:34.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.909+0000 7f47f1085700 1 --2- 172.21.15.129:0/2837856214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47ec103970 0x7f47ec074710 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f47e40062a0 tx=0x7f47e4009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:34.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.909+0000 7f47e9ffb700 1 -- 172.21.15.129:0/2837856214 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e4021070 con 0x7f47ec103970 2024-03-11T20:51:34.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.909+0000 7f47f2ae8700 1 -- 172.21.15.129:0/2837856214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47ec1a14a0 con 0x7f47ec103970 2024-03-11T20:51:34.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.909+0000 7f47e9ffb700 1 -- 172.21.15.129:0/2837856214 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f47e4007b80 con 0x7f47ec103970 2024-03-11T20:51:34.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.909+0000 7f47e9ffb700 1 -- 172.21.15.129:0/2837856214 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e4012650 con 0x7f47ec103970 2024-03-11T20:51:34.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.909+0000 7f47f2ae8700 1 -- 172.21.15.129:0/2837856214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47ec1a1950 con 0x7f47ec103970 2024-03-11T20:51:34.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.910+0000 7f47f2ae8700 1 -- 172.21.15.129:0/2837856214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47ec0502f0 con 0x7f47ec103970 2024-03-11T20:51:34.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.911+0000 7f47e9ffb700 1 -- 172.21.15.129:0/2837856214 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) v1 ==== 91806+0+0 (secure 0 0 0) 0x7f47e4024070 con 0x7f47ec103970 2024-03-11T20:51:34.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.911+0000 7f47e9ffb700 1 --2- 172.21.15.129:0/2837856214 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f47dc06ded0 0x7f47dc070390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:34.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.911+0000 7f47e9ffb700 1 -- 172.21.15.129:0/2837856214 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7207+0+0 (secure 0 0 0) 0x7f47e400f020 con 0x7f47ec103970 2024-03-11T20:51:34.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.911+0000 7f47f0884700 1 --2- 172.21.15.129:0/2837856214 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f47dc06ded0 0x7f47dc070390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:34.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.912+0000 7f47f0884700 1 --2- 172.21.15.129:0/2837856214 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f47dc06ded0 0x7f47dc070390 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f47e0009680 tx=0x7f47e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:34.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:34.916+0000 7f47e9ffb700 1 -- 172.21.15.129:0/2837856214 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f47e402b020 con 0x7f47ec103970 2024-03-11T20:51:35.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:35.194+0000 7f47f2ae8700 1 -- 172.21.15.129:0/2837856214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f47ec02d830 con 0x7f47ec103970 2024-03-11T20:51:35.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:35.198+0000 7f47e9ffb700 1 -- 172.21.15.129:0/2837856214 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f47e405c820 con 0x7f47ec103970 2024-03-11T20:51:35.194 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-03-11T20:51:35.195 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQDXbu9lbTmwCxAAE/EpzEURVkU9J0KCT3q/DQ== 2024-03-11T20:51:35.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:35.201+0000 7f47f2ae8700 1 -- 172.21.15.129:0/2837856214 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f47dc06ded0 msgr2=0x7f47dc070390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:35.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:35.201+0000 7f47f2ae8700 1 --2- 172.21.15.129:0/2837856214 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f47dc06ded0 0x7f47dc070390 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f47e0009680 tx=0x7f47e0009260 comp rx=0 tx=0).stop 2024-03-11T20:51:35.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:35.201+0000 7f47f2ae8700 1 -- 172.21.15.129:0/2837856214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47ec103970 msgr2=0x7f47ec074710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:35.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:35.201+0000 7f47f2ae8700 1 --2- 172.21.15.129:0/2837856214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47ec103970 0x7f47ec074710 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f47e40062a0 tx=0x7f47e4009000 comp rx=0 tx=0).stop 2024-03-11T20:51:35.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:35.201+0000 7f47f2ae8700 1 -- 172.21.15.129:0/2837856214 shutdown_connections 2024-03-11T20:51:35.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:35.201+0000 7f47f2ae8700 1 --2- 172.21.15.129:0/2837856214 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f47dc06ded0 0x7f47dc070390 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:35.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:35.201+0000 7f47f2ae8700 1 --2- 172.21.15.129:0/2837856214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47ec103970 0x7f47ec074710 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:35.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:35.201+0000 7f47f2ae8700 1 --2- 172.21.15.129:0/2837856214 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f47ec101e50 0x7f47ec075b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:35.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:35.201+0000 7f47f2ae8700 1 --2- 172.21.15.129:0/2837856214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47ec102fb0 0x7f47ec0741d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:35.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:35.201+0000 7f47f2ae8700 1 -- 172.21.15.129:0/2837856214 >> 172.21.15.129:0/2837856214 conn(0x7f47ec0fd3f0 msgr2=0x7f47ec0feef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:35.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:35.201+0000 7f47f2ae8700 1 -- 172.21.15.129:0/2837856214 shutdown_connections 2024-03-11T20:51:35.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:35.201+0000 7f47f2ae8700 1 -- 172.21.15.129:0/2837856214 wait complete. 2024-03-11T20:51:35.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:34 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:35.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:34 smithi157 ceph-mon[33359]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s; 13 B/s, 0 objects/s recovering 2024-03-11T20:51:35.924 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2837856214' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:35.924 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2837856214' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T20:51:35.924 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2837856214' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:35.924 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2837856214' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T20:51:35.952 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:51:35.952 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T20:51:35.952 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T20:51:35.995 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T20:51:36.019 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2837856214' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:36.019 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2837856214' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T20:51:36.240 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T20:51:36.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:36 smithi157 ceph-mon[33359]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 41 B/s, 1 objects/s recovering 2024-03-11T20:51:37.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:36 smithi129 ceph-mon[30184]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 41 B/s, 1 objects/s recovering 2024-03-11T20:51:37.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:36 smithi129 ceph-mon[42066]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 41 B/s, 1 objects/s recovering 2024-03-11T20:51:38.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.003+0000 7f890a1fe700 1 -- 172.21.15.157:0/2967673731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89041016a0 msgr2=0x7f8904101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:38.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.003+0000 7f890a1fe700 1 --2- 172.21.15.157:0/2967673731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89041016a0 0x7f8904101b00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f88f80099c0 tx=0x7f88f8009cd0 comp rx=0 tx=0).stop 2024-03-11T20:51:38.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.003+0000 7f890a1fe700 1 -- 172.21.15.157:0/2967673731 shutdown_connections 2024-03-11T20:51:38.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.003+0000 7f890a1fe700 1 --2- 172.21.15.157:0/2967673731 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8904102040 0x7f89041088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:38.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.003+0000 7f890a1fe700 1 --2- 172.21.15.157:0/2967673731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89041004a0 0x7f89041008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:38.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.003+0000 7f890a1fe700 1 --2- 172.21.15.157:0/2967673731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89041016a0 0x7f8904101b00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:38.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.003+0000 7f890a1fe700 1 -- 172.21.15.157:0/2967673731 >> 172.21.15.157:0/2967673731 conn(0x7f89040fba80 msgr2=0x7f89040fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:38.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.003+0000 7f890a1fe700 1 -- 172.21.15.157:0/2967673731 shutdown_connections 2024-03-11T20:51:38.001 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.003+0000 7f890a1fe700 1 -- 172.21.15.157:0/2967673731 wait complete. 2024-03-11T20:51:38.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.004+0000 7f890a1fe700 1 Processor -- start 2024-03-11T20:51:38.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.004+0000 7f890a1fe700 1 -- start start 2024-03-11T20:51:38.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.004+0000 7f890a1fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89041004a0 0x7f8904195290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:38.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.004+0000 7f890a1fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89041016a0 0x7f89041957d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:38.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.004+0000 7f890a1fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8904102040 0x7f890419c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:38.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.004+0000 7f890a1fe700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890410ace0 con 0x7f89041004a0 2024-03-11T20:51:38.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.004+0000 7f890a1fe700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890410ab60 con 0x7f89041016a0 2024-03-11T20:51:38.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.004+0000 7f890a1fe700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890410ae60 con 0x7f8904102040 2024-03-11T20:51:38.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.004+0000 7f8902ffd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89041016a0 0x7f89041957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:38.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.004+0000 7f8902ffd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89041016a0 0x7f89041957d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:42748/0 (socket says 172.21.15.157:42748) 2024-03-11T20:51:38.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.004+0000 7f8902ffd700 1 -- 172.21.15.157:0/843433038 learned_addr learned my addr 172.21.15.157:0/843433038 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-11T20:51:38.002 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.004+0000 7f89037fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89041004a0 0x7f8904195290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:38.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.004+0000 7f8903fff700 1 --2- 172.21.15.157:0/843433038 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8904102040 0x7f890419c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:38.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.005+0000 7f89037fe700 1 -- 172.21.15.157:0/843433038 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8904102040 msgr2=0x7f890419c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:38.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.005+0000 7f89037fe700 1 --2- 172.21.15.157:0/843433038 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8904102040 0x7f890419c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:38.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.005+0000 7f89037fe700 1 -- 172.21.15.157:0/843433038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89041016a0 msgr2=0x7f89041957d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:38.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.005+0000 7f89037fe700 1 --2- 172.21.15.157:0/843433038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89041016a0 0x7f89041957d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:38.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.005+0000 7f89037fe700 1 -- 172.21.15.157:0/843433038 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890419cef0 con 0x7f89041004a0 2024-03-11T20:51:38.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.005+0000 7f8902ffd700 1 --2- 172.21.15.157:0/843433038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89041016a0 0x7f89041957d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:51:38.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.005+0000 7f89037fe700 1 --2- 172.21.15.157:0/843433038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89041004a0 0x7f8904195290 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f88f400e9f0 tx=0x7f88f400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:38.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.005+0000 7f8900ff9700 1 -- 172.21.15.157:0/843433038 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f4004220 con 0x7f89041004a0 2024-03-11T20:51:38.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.005+0000 7f890a1fe700 1 -- 172.21.15.157:0/843433038 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890419d1e0 con 0x7f89041004a0 2024-03-11T20:51:38.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.005+0000 7f8900ff9700 1 -- 172.21.15.157:0/843433038 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f88f40043c0 con 0x7f89041004a0 2024-03-11T20:51:38.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.005+0000 7f8900ff9700 1 -- 172.21.15.157:0/843433038 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f4003b00 con 0x7f89041004a0 2024-03-11T20:51:38.003 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.005+0000 7f890a1fe700 1 -- 172.21.15.157:0/843433038 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f890419d720 con 0x7f89041004a0 2024-03-11T20:51:38.004 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.006+0000 7f890a1fe700 1 -- 172.21.15.157:0/843433038 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89040502f0 con 0x7f89041004a0 2024-03-11T20:51:38.004 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.007+0000 7f8900ff9700 1 -- 172.21.15.157:0/843433038 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) v1 ==== 91806+0+0 (secure 0 0 0) 0x7f88f4003ca0 con 0x7f89041004a0 2024-03-11T20:51:38.004 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.007+0000 7f8900ff9700 1 --2- 172.21.15.157:0/843433038 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f88ec06ded0 0x7f88ec070390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:38.005 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.007+0000 7f8900ff9700 1 -- 172.21.15.157:0/843433038 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7207+0+0 (secure 0 0 0) 0x7f88f408e320 con 0x7f89041004a0 2024-03-11T20:51:38.005 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.007+0000 7f8902ffd700 1 --2- 172.21.15.157:0/843433038 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f88ec06ded0 0x7f88ec070390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:38.005 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.008+0000 7f8902ffd700 1 --2- 172.21.15.157:0/843433038 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f88ec06ded0 0x7f88ec070390 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009750 tx=0x7f88f8003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:38.008 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.013+0000 7f8900ff9700 1 -- 172.21.15.157:0/843433038 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f88f405b830 con 0x7f89041004a0 2024-03-11T20:51:38.297 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.301+0000 7f890a1fe700 1 -- 172.21.15.157:0/843433038 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f890402d830 con 0x7f89041004a0 2024-03-11T20:51:38.299 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.303+0000 7f8900ff9700 1 -- 172.21.15.157:0/843433038 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f88f405b130 con 0x7f89041004a0 2024-03-11T20:51:38.299 INFO:teuthology.orchestra.run.smithi157.stdout:[client.1] 2024-03-11T20:51:38.299 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDabu9lR8AJEhAAtxPlk6NReoP/i8Fppnhq7w== 2024-03-11T20:51:38.301 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.305+0000 7f890a1fe700 1 -- 172.21.15.157:0/843433038 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f88ec06ded0 msgr2=0x7f88ec070390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:38.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.305+0000 7f890a1fe700 1 --2- 172.21.15.157:0/843433038 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f88ec06ded0 0x7f88ec070390 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009750 tx=0x7f88f8003680 comp rx=0 tx=0).stop 2024-03-11T20:51:38.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.306+0000 7f890a1fe700 1 -- 172.21.15.157:0/843433038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89041004a0 msgr2=0x7f8904195290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:38.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.306+0000 7f890a1fe700 1 --2- 172.21.15.157:0/843433038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89041004a0 0x7f8904195290 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f88f400e9f0 tx=0x7f88f400ed00 comp rx=0 tx=0).stop 2024-03-11T20:51:38.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.306+0000 7f890a1fe700 1 -- 172.21.15.157:0/843433038 shutdown_connections 2024-03-11T20:51:38.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.306+0000 7f890a1fe700 1 --2- 172.21.15.157:0/843433038 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f88ec06ded0 0x7f88ec070390 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:38.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.306+0000 7f890a1fe700 1 --2- 172.21.15.157:0/843433038 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8904102040 0x7f890419c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:38.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.306+0000 7f890a1fe700 1 --2- 172.21.15.157:0/843433038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89041004a0 0x7f8904195290 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:38.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.306+0000 7f890a1fe700 1 --2- 172.21.15.157:0/843433038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89041016a0 0x7f89041957d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:38.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.306+0000 7f890a1fe700 1 -- 172.21.15.157:0/843433038 >> 172.21.15.157:0/843433038 conn(0x7f89040fba80 msgr2=0x7f8904104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:38.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.306+0000 7f890a1fe700 1 -- 172.21.15.157:0/843433038 shutdown_connections 2024-03-11T20:51:38.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-11T20:51:38.306+0000 7f890a1fe700 1 -- 172.21.15.157:0/843433038 wait complete. 2024-03-11T20:51:38.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:38 smithi157 ceph-mon[33359]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-03-11T20:51:38.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.157:0/843433038' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:38.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.157:0/843433038' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T20:51:39.008 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-11T20:51:39.008 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-11T20:51:39.008 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-11T20:51:39.090 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T20:51:39.090 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T20:51:39.090 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T20:51:39.118 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:38 smithi129 ceph-mon[30184]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-03-11T20:51:39.118 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.157:0/843433038' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:39.118 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.157:0/843433038' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T20:51:39.118 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:38 smithi129 ceph-mon[42066]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-03-11T20:51:39.119 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.157:0/843433038' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T20:51:39.119 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.157:0/843433038' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T20:51:39.339 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:51:40.936 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:40 smithi129 ceph-mon[30184]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 35 B/s, 1 objects/s recovering 2024-03-11T20:51:40.936 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:40 smithi129 ceph-mon[42066]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 35 B/s, 1 objects/s recovering 2024-03-11T20:51:41.038 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:40 smithi157 ceph-mon[33359]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 35 B/s, 1 objects/s recovering 2024-03-11T20:51:41.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.183+0000 7f36fba51700 1 -- 172.21.15.129:0/632185946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36f4107100 msgr2=0x7f36f410ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:41.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.183+0000 7f36fba51700 1 --2- 172.21.15.129:0/632185946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36f4107100 0x7f36f410ecd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f36ec00b0d0 tx=0x7f36ec00b3e0 comp rx=0 tx=0).stop 2024-03-11T20:51:41.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.183+0000 7f36fba51700 1 -- 172.21.15.129:0/632185946 shutdown_connections 2024-03-11T20:51:41.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.183+0000 7f36fba51700 1 --2- 172.21.15.129:0/632185946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36f4107100 0x7f36f410ecd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:41.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.183+0000 7f36fba51700 1 --2- 172.21.15.129:0/632185946 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36f4102020 0x7f36f4102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:41.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.183+0000 7f36fba51700 1 --2- 172.21.15.129:0/632185946 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36f41029d0 0x7f36f4106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:41.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.183+0000 7f36fba51700 1 -- 172.21.15.129:0/632185946 >> 172.21.15.129:0/632185946 conn(0x7f36f40fd890 msgr2=0x7f36f40ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:41.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.183+0000 7f36fba51700 1 -- 172.21.15.129:0/632185946 shutdown_connections 2024-03-11T20:51:41.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.183+0000 7f36fba51700 1 -- 172.21.15.129:0/632185946 wait complete. 2024-03-11T20:51:41.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36fba51700 1 Processor -- start 2024-03-11T20:51:41.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36fba51700 1 -- start start 2024-03-11T20:51:41.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36fba51700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36f4102020 0x7f36f419d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:41.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36fba51700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36f41029d0 0x7f36f419dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:41.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36fba51700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36f4107100 0x7f36f41978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:41.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36fba51700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f4111610 con 0x7f36f4102020 2024-03-11T20:51:41.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36fba51700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f4111490 con 0x7f36f4107100 2024-03-11T20:51:41.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36fba51700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f4111790 con 0x7f36f41029d0 2024-03-11T20:51:41.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36f97ed700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36f4102020 0x7f36f419d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:41.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36f8fec700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36f41029d0 0x7f36f419dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:41.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36f8fec700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36f41029d0 0x7f36f419dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:38420/0 (socket says 172.21.15.129:38420) 2024-03-11T20:51:41.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36f8fec700 1 -- 172.21.15.129:0/1174968250 learned_addr learned my addr 172.21.15.129:0/1174968250 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:51:41.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36f9fee700 1 --2- 172.21.15.129:0/1174968250 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36f4107100 0x7f36f41978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:41.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36f97ed700 1 -- 172.21.15.129:0/1174968250 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36f41029d0 msgr2=0x7f36f419dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:41.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36f97ed700 1 --2- 172.21.15.129:0/1174968250 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36f41029d0 0x7f36f419dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:41.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36f97ed700 1 -- 172.21.15.129:0/1174968250 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36f4107100 msgr2=0x7f36f41978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:41.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36f97ed700 1 --2- 172.21.15.129:0/1174968250 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36f4107100 0x7f36f41978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:41.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36f97ed700 1 -- 172.21.15.129:0/1174968250 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f4198030 con 0x7f36f4102020 2024-03-11T20:51:41.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.184+0000 7f36f8fec700 1 --2- 172.21.15.129:0/1174968250 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36f41029d0 0x7f36f419dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:51:41.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.185+0000 7f36f97ed700 1 --2- 172.21.15.129:0/1174968250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36f4102020 0x7f36f419d6c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f36e400e9f0 tx=0x7f36e400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:41.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.185+0000 7f36ea7fc700 1 -- 172.21.15.129:0/1174968250 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e400cb70 con 0x7f36f4102020 2024-03-11T20:51:41.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.185+0000 7f36ea7fc700 1 -- 172.21.15.129:0/1174968250 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f36e400ee90 con 0x7f36f4102020 2024-03-11T20:51:41.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.185+0000 7f36fba51700 1 -- 172.21.15.129:0/1174968250 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f4198320 con 0x7f36f4102020 2024-03-11T20:51:41.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.185+0000 7f36ea7fc700 1 -- 172.21.15.129:0/1174968250 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e4003c10 con 0x7f36f4102020 2024-03-11T20:51:41.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.185+0000 7f36fba51700 1 -- 172.21.15.129:0/1174968250 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f41a4590 con 0x7f36f4102020 2024-03-11T20:51:41.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.186+0000 7f36fba51700 1 -- 172.21.15.129:0/1174968250 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f4068aa0 con 0x7f36f4102020 2024-03-11T20:51:41.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.186+0000 7f36ea7fc700 1 -- 172.21.15.129:0/1174968250 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 16) v1 ==== 91806+0+0 (secure 0 0 0) 0x7f36e4003db0 con 0x7f36f4102020 2024-03-11T20:51:41.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.187+0000 7f36ea7fc700 1 --2- 172.21.15.129:0/1174968250 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f36e006de20 0x7f36e00702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:41.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.187+0000 7f36ea7fc700 1 -- 172.21.15.129:0/1174968250 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7207+0+0 (secure 0 0 0) 0x7f36e408e1e0 con 0x7f36f4102020 2024-03-11T20:51:41.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.187+0000 7f36f8fec700 1 --2- 172.21.15.129:0/1174968250 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f36e006de20 0x7f36e00702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:41.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.187+0000 7f36f8fec700 1 --2- 172.21.15.129:0/1174968250 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f36e006de20 0x7f36e00702e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f36f4108670 tx=0x7f36f000b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:41.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.191+0000 7f36ea7fc700 1 -- 172.21.15.129:0/1174968250 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f36e405b770 con 0x7f36f4102020 2024-03-11T20:51:41.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.439+0000 7f36fba51700 1 -- 172.21.15.129:0/1174968250 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f36f402d830 con 0x7f36f4102020 2024-03-11T20:51:41.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.442+0000 7f36ea7fc700 1 -- 172.21.15.129:0/1174968250 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v16) v1 ==== 74+0+175653 (secure 0 0 0) 0x7f36e405b070 con 0x7f36f4102020 2024-03-11T20:51:41.441 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:51:41.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.446+0000 7f36fba51700 1 -- 172.21.15.129:0/1174968250 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f36e006de20 msgr2=0x7f36e00702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:41.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.446+0000 7f36fba51700 1 --2- 172.21.15.129:0/1174968250 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f36e006de20 0x7f36e00702e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f36f4108670 tx=0x7f36f000b480 comp rx=0 tx=0).stop 2024-03-11T20:51:41.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.446+0000 7f36fba51700 1 -- 172.21.15.129:0/1174968250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36f4102020 msgr2=0x7f36f419d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:41.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.446+0000 7f36fba51700 1 --2- 172.21.15.129:0/1174968250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36f4102020 0x7f36f419d6c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f36e400e9f0 tx=0x7f36e400ed00 comp rx=0 tx=0).stop 2024-03-11T20:51:41.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.446+0000 7f36fba51700 1 -- 172.21.15.129:0/1174968250 shutdown_connections 2024-03-11T20:51:41.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.446+0000 7f36fba51700 1 --2- 172.21.15.129:0/1174968250 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f36e006de20 0x7f36e00702e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:41.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.446+0000 7f36fba51700 1 --2- 172.21.15.129:0/1174968250 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36f4107100 0x7f36f41978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:41.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.446+0000 7f36fba51700 1 --2- 172.21.15.129:0/1174968250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36f4102020 0x7f36f419d6c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:41.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.446+0000 7f36fba51700 1 --2- 172.21.15.129:0/1174968250 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36f41029d0 0x7f36f419dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:41.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.446+0000 7f36fba51700 1 -- 172.21.15.129:0/1174968250 >> 172.21.15.129:0/1174968250 conn(0x7f36f40fd890 msgr2=0x7f36f40ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:41.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.446+0000 7f36fba51700 1 -- 172.21.15.129:0/1174968250 shutdown_connections 2024-03-11T20:51:41.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:41.446+0000 7f36fba51700 1 -- 172.21.15.129:0/1174968250 wait complete. 2024-03-11T20:51:42.037 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:41 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1174968250' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T20:51:42.038 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:41 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1174968250' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T20:51:42.062 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":2866184010},{"type":"v1","addr":"172.21.15.129:6801","nonce":2866184010}]},"active_addr":"172.21.15.129:6801/2866184010","active_change":"2024-03-11T20:46:29.573879+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.129:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":1624357520}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":1962558927}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":2937681946}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":2530444419}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.129:0","nonce":3331491924}]}]} 2024-03-11T20:51:42.067 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T20:51:42.067 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T20:51:42.067 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T20:51:42.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:41 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1174968250' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T20:51:42.339 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:51:43.159 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:42 smithi157 ceph-mon[33359]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 32 B/s, 1 objects/s recovering 2024-03-11T20:51:43.185 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:42 smithi129 ceph-mon[30184]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 32 B/s, 1 objects/s recovering 2024-03-11T20:51:43.185 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:42 smithi129 ceph-mon[42066]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 32 B/s, 1 objects/s recovering 2024-03-11T20:51:43.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:51:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:51:44.097 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:43 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:44.097 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:43 smithi129 ceph-mon[30184]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-11T20:51:44.097 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:43 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:44.097 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:43 smithi129 ceph-mon[42066]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-11T20:51:44.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.111+0000 7f36dae97700 1 -- 172.21.15.129:0/3555248257 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36d4105d20 msgr2=0x7f36d4106100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:44.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.111+0000 7f36dae97700 1 --2- 172.21.15.129:0/3555248257 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36d4105d20 0x7f36d4106100 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f36c8009990 tx=0x7f36c8009ca0 comp rx=0 tx=0).stop 2024-03-11T20:51:44.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.111+0000 7f36dae97700 1 -- 172.21.15.129:0/3555248257 shutdown_connections 2024-03-11T20:51:44.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.111+0000 7f36dae97700 1 --2- 172.21.15.129:0/3555248257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36d41021f0 0x7f36d410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:44.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.111+0000 7f36dae97700 1 --2- 172.21.15.129:0/3555248257 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36d4105d20 0x7f36d4106100 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:44.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.111+0000 7f36dae97700 1 --2- 172.21.15.129:0/3555248257 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36d4101830 0x7f36d4101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:44.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.111+0000 7f36dae97700 1 -- 172.21.15.129:0/3555248257 >> 172.21.15.129:0/3555248257 conn(0x7f36d4076020 msgr2=0x7f36d4076430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:44.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.111+0000 7f36dae97700 1 -- 172.21.15.129:0/3555248257 shutdown_connections 2024-03-11T20:51:44.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.111+0000 7f36dae97700 1 -- 172.21.15.129:0/3555248257 wait complete. 2024-03-11T20:51:44.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.112+0000 7f36dae97700 1 Processor -- start 2024-03-11T20:51:44.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.112+0000 7f36dae97700 1 -- start start 2024-03-11T20:51:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.112+0000 7f36dae97700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36d4101830 0x7f36d419d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.112+0000 7f36dae97700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36d41021f0 0x7f36d419dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.112+0000 7f36dae97700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36d4105d20 0x7f36d4197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.112+0000 7f36dae97700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d4110560 con 0x7f36d4101830 2024-03-11T20:51:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.112+0000 7f36dae97700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d41103e0 con 0x7f36d4105d20 2024-03-11T20:51:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.112+0000 7f36dae97700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d41106e0 con 0x7f36d41021f0 2024-03-11T20:51:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.112+0000 7f36d8c33700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36d4101830 0x7f36d419d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.112+0000 7f36d3fff700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36d41021f0 0x7f36d419dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.112+0000 7f36d8c33700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36d4101830 0x7f36d419d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46816/0 (socket says 172.21.15.129:46816) 2024-03-11T20:51:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.112+0000 7f36d8c33700 1 -- 172.21.15.129:0/1484757324 learned_addr learned my addr 172.21.15.129:0/1484757324 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:51:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.112+0000 7f36d9434700 1 --2- 172.21.15.129:0/1484757324 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36d4105d20 0x7f36d4197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.113+0000 7f36d3fff700 1 -- 172.21.15.129:0/1484757324 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36d4105d20 msgr2=0x7f36d4197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.113+0000 7f36d3fff700 1 --2- 172.21.15.129:0/1484757324 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36d4105d20 0x7f36d4197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.113+0000 7f36d3fff700 1 -- 172.21.15.129:0/1484757324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36d4101830 msgr2=0x7f36d419d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.113+0000 7f36d3fff700 1 --2- 172.21.15.129:0/1484757324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36d4101830 0x7f36d419d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.113+0000 7f36d3fff700 1 -- 172.21.15.129:0/1484757324 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d41980c0 con 0x7f36d41021f0 2024-03-11T20:51:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.113+0000 7f36d9434700 1 --2- 172.21.15.129:0/1484757324 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36d4105d20 0x7f36d4197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T20:51:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.113+0000 7f36d3fff700 1 --2- 172.21.15.129:0/1484757324 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36d41021f0 0x7f36d419dce0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f36c000c970 tx=0x7f36c000cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.113+0000 7f36d1ffb700 1 -- 172.21.15.129:0/1484757324 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c000ce60 con 0x7f36d41021f0 2024-03-11T20:51:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.113+0000 7f36d1ffb700 1 -- 172.21.15.129:0/1484757324 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f36c0004830 con 0x7f36d41021f0 2024-03-11T20:51:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.113+0000 7f36dae97700 1 -- 172.21.15.129:0/1484757324 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d41983b0 con 0x7f36d41021f0 2024-03-11T20:51:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.113+0000 7f36d1ffb700 1 -- 172.21.15.129:0/1484757324 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0005320 con 0x7f36d41021f0 2024-03-11T20:51:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.113+0000 7f36dae97700 1 -- 172.21.15.129:0/1484757324 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d41a4670 con 0x7f36d41021f0 2024-03-11T20:51:44.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.114+0000 7f36dae97700 1 -- 172.21.15.129:0/1484757324 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d4068aa0 con 0x7f36d41021f0 2024-03-11T20:51:44.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.115+0000 7f36d1ffb700 1 -- 172.21.15.129:0/1484757324 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 16) v1 ==== 91806+0+0 (secure 0 0 0) 0x7f36c0005500 con 0x7f36d41021f0 2024-03-11T20:51:44.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.115+0000 7f36d1ffb700 1 --2- 172.21.15.129:0/1484757324 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f36c406de20 0x7f36c40702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:44.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.115+0000 7f36d1ffb700 1 -- 172.21.15.129:0/1484757324 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7207+0+0 (secure 0 0 0) 0x7f36c008dc00 con 0x7f36d41021f0 2024-03-11T20:51:44.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.115+0000 7f36d8c33700 1 --2- 172.21.15.129:0/1484757324 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f36c406de20 0x7f36c40702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:44.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.115+0000 7f36d8c33700 1 --2- 172.21.15.129:0/1484757324 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f36c406de20 0x7f36c40702e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f36c8006010 tx=0x7f36c801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:44.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.120+0000 7f36d1ffb700 1 -- 172.21.15.129:0/1484757324 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f36c0093020 con 0x7f36d41021f0 2024-03-11T20:51:44.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:43 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:51:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:43 smithi157 ceph-mon[33359]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-11T20:51:44.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.312+0000 7f36dae97700 1 -- 172.21.15.129:0/1484757324 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36d41a4900 con 0x7f36d41021f0 2024-03-11T20:51:44.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.313+0000 7f36d1ffb700 1 -- 172.21.15.129:0/1484757324 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+22809 (secure 0 0 0) 0x7f36c0014030 con 0x7f36d41021f0 2024-03-11T20:51:44.310 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:51:44.310 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"fsid":"27bbebfa-dfe8-11ee-95c8-87774f69a715","created":"2024-03-11T20:45:35.170893+0000","modified":"2024-03-11T20:51:31.742907+0000","last_up_change":"2024-03-11T20:50:57.449832+0000","last_in_change":"2024-03-11T20:50:39.277288+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T20:48:49.904657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-11T20:51:09.381905+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-11T20:51:10.107130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-11T20:51:12.032362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-11T20:51:13.974387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T20:51:16.043073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5dbe3ee2-8c30-4242-baf0-a36e1ac0be8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3618360833},{"type":"v1","addr":"172.21.15.129:6803","nonce":3618360833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3618360833},{"type":"v1","addr":"172.21.15.129:6805","nonce":3618360833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":3618360833},{"type":"v1","addr":"172.21.15.129:6809","nonce":3618360833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3618360833},{"type":"v1","addr":"172.21.15.129:6807","nonce":3618360833}]},"public_addr":"172.21.15.129:6803/3618360833","cluster_addr":"172.21.15.129:6805/3618360833","heartbeat_back_addr":"172.21.15.129:6809/3618360833","heartbeat_front_addr":"172.21.15.129:6807/3618360833","state":["exists","up"]},{"osd":1,"uuid":"d5495628-e325-4a58-b682-50a26ef780fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2519234527},{"type":"v1","addr":"172.21.15.129:6811","nonce":2519234527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2519234527},{"type":"v1","addr":"172.21.15.129:6813","nonce":2519234527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":2519234527},{"type":"v1","addr":"172.21.15.129:6817","nonce":2519234527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2519234527},{"type":"v1","addr":"172.21.15.129:6815","nonce":2519234527}]},"public_addr":"172.21.15.129:6811/2519234527","cluster_addr":"172.21.15.129:6813/2519234527","heartbeat_back_addr":"172.21.15.129:6817/2519234527","heartbeat_front_addr":"172.21.15.129:6815/2519234527","state":["exists","up"]},{"osd":2,"uuid":"2b6db120-1132-4369-9f17-777545ad4a07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1600233087},{"type":"v1","addr":"172.21.15.129:6819","nonce":1600233087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1600233087},{"type":"v1","addr":"172.21.15.129:6821","nonce":1600233087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1600233087},{"type":"v1","addr":"172.21.15.129:6825","nonce":1600233087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1600233087},{"type":"v1","addr":"172.21.15.129:6823","nonce":1600233087}]},"public_addr":"172.21.15.129:6819/1600233087","cluster_addr":"172.21.15.129:6821/1600233087","heartbeat_back_addr":"172.21.15.129:6825/1600233087","heartbeat_front_addr":"172.21.15.129:6823/1600233087","state":["exists","up"]},{"osd":3,"uuid":"d9d3db4d-385d-4745-b71f-0796ff551f4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2265856698},{"type":"v1","addr":"172.21.15.129:6827","nonce":2265856698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2265856698},{"type":"v1","addr":"172.21.15.129:6829","nonce":2265856698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2265856698},{"type":"v1","addr":"172.21.15.129:6833","nonce":2265856698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2265856698},{"type":"v1","addr":"172.21.15.129:6831","nonce":2265856698}]},"public_addr":"172.21.15.129:6827/2265856698","cluster_addr":"172.21.15.129:6829/2265856698","heartbeat_back_addr":"172.21.15.129:6833/2265856698","heartbeat_front_addr":"172.21.15.129:6831/2265856698","state":["exists","up"]},{"osd":4,"uuid":"2aa92e2f-5687-4260-985a-3006da3cc803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3991533346},{"type":"v1","addr":"172.21.15.157:6801","nonce":3991533346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3991533346},{"type":"v1","addr":"172.21.15.157:6803","nonce":3991533346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3991533346},{"type":"v1","addr":"172.21.15.157:6807","nonce":3991533346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3991533346},{"type":"v1","addr":"172.21.15.157:6805","nonce":3991533346}]},"public_addr":"172.21.15.157:6801/3991533346","cluster_addr":"172.21.15.157:6803/3991533346","heartbeat_back_addr":"172.21.15.157:6807/3991533346","heartbeat_front_addr":"172.21.15.157:6805/3991533346","state":["exists","up"]},{"osd":5,"uuid":"dac5fa60-f01b-4323-8596-cc0f3a5030c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":2444017632},{"type":"v1","addr":"172.21.15.157:6809","nonce":2444017632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":2444017632},{"type":"v1","addr":"172.21.15.157:6811","nonce":2444017632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":2444017632},{"type":"v1","addr":"172.21.15.157:6815","nonce":2444017632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":2444017632},{"type":"v1","addr":"172.21.15.157:6813","nonce":2444017632}]},"public_addr":"172.21.15.157:6809/2444017632","cluster_addr":"172.21.15.157:6811/2444017632","heartbeat_back_addr":"172.21.15.157:6815/2444017632","heartbeat_front_addr":"172.21.15.157:6813/2444017632","state":["exists","up"]},{"osd":6,"uuid":"9c489abd-5f6c-4fb9-954a-aabd409fffd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":2710731257},{"type":"v1","addr":"172.21.15.157:6817","nonce":2710731257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":2710731257},{"type":"v1","addr":"172.21.15.157:6819","nonce":2710731257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":2710731257},{"type":"v1","addr":"172.21.15.157:6823","nonce":2710731257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":2710731257},{"type":"v1","addr":"172.21.15.157:6821","nonce":2710731257}]},"public_addr":"172.21.15.157:6817/2710731257","cluster_addr":"172.21.15.157:6819/2710731257","heartbeat_back_addr":"172.21.15.157:6823/2710731257","heartbeat_front_addr":"172.21.15.157:6821/2710731257","state":["exists","up"]},{"osd":7,"uuid":"22bacf1f-35c5-4dee-81a6-92dff2f076c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1786686696},{"type":"v1","addr":"172.21.15.157:6825","nonce":1786686696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1786686696},{"type":"v1","addr":"172.21.15.157:6827","nonce":1786686696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1786686696},{"type":"v1","addr":"172.21.15.157:6831","nonce":1786686696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1786686696},{"type":"v1","addr":"172.21.15.157:6829","nonce":1786686696}]},"public_addr":"172.21.15.157:6825/1786686696","cluster_addr":"172.21.15.157:6827/1786686696","heartbeat_back_addr":"172.21.15.157:6831/1786686696","heartbeat_front_addr":"172.21.15.157:6829/1786686696","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:47:54.931014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:48:20.791637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:48:46.661578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:49:13.177044+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:49:37.603176+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:50:03.465298+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:50:29.681381+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:50:55.559486+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2734363055":"2024-03-12T20:46:29.573760+0000","172.21.15.129:0/1185914521":"2024-03-12T20:46:29.573760+0000","172.21.15.129:6800/2213467472":"2024-03-12T20:46:02.671885+0000","172.21.15.129:0/1229097270":"2024-03-12T20:46:02.671885+0000","172.21.15.129:6801/521270824":"2024-03-12T20:46:29.573760+0000","172.21.15.129:6801/2213467472":"2024-03-12T20:46:02.671885+0000","172.21.15.129:0/160155772":"2024-03-12T20:46:29.573760+0000","172.21.15.129:0/4048407456":"2024-03-12T20:46:02.671885+0000","172.21.15.129:0/1059875431":"2024-03-12T20:46:02.671885+0000","172.21.15.129:6800/521270824":"2024-03-12T20:46:29.573760+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T20:51:44.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.316+0000 7f36dae97700 1 -- 172.21.15.129:0/1484757324 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f36c406de20 msgr2=0x7f36c40702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:44.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.316+0000 7f36dae97700 1 --2- 172.21.15.129:0/1484757324 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f36c406de20 0x7f36c40702e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f36c8006010 tx=0x7f36c801a040 comp rx=0 tx=0).stop 2024-03-11T20:51:44.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.316+0000 7f36dae97700 1 -- 172.21.15.129:0/1484757324 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36d41021f0 msgr2=0x7f36d419dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:44.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.316+0000 7f36dae97700 1 --2- 172.21.15.129:0/1484757324 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36d41021f0 0x7f36d419dce0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f36c000c970 tx=0x7f36c000cc80 comp rx=0 tx=0).stop 2024-03-11T20:51:44.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.316+0000 7f36dae97700 1 -- 172.21.15.129:0/1484757324 shutdown_connections 2024-03-11T20:51:44.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.316+0000 7f36dae97700 1 --2- 172.21.15.129:0/1484757324 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f36c406de20 0x7f36c40702e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:44.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.316+0000 7f36dae97700 1 --2- 172.21.15.129:0/1484757324 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36d4105d20 0x7f36d4197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:44.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.316+0000 7f36dae97700 1 --2- 172.21.15.129:0/1484757324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36d4101830 0x7f36d419d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:44.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.316+0000 7f36dae97700 1 --2- 172.21.15.129:0/1484757324 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f36d41021f0 0x7f36d419dce0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:44.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.316+0000 7f36dae97700 1 -- 172.21.15.129:0/1484757324 >> 172.21.15.129:0/1484757324 conn(0x7f36d4076020 msgr2=0x7f36d4104f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:44.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.316+0000 7f36dae97700 1 -- 172.21.15.129:0/1484757324 shutdown_connections 2024-03-11T20:51:44.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:44.316+0000 7f36dae97700 1 -- 172.21.15.129:0/1484757324 wait complete. 2024-03-11T20:51:45.085 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1484757324' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:51:45.085 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1484757324' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:51:45.115 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T20:51:45.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T20:51:45.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1484757324' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:51:45.353 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:51:46.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:45 smithi129 ceph-mon[42066]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-11T20:51:46.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:45 smithi129 ceph-mon[30184]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-11T20:51:46.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:45 smithi157 ceph-mon[33359]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-11T20:51:47.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.193+0000 7f91899c6700 1 -- 172.21.15.129:0/1299815680 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f918406a760 msgr2=0x7f918406ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:47.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.193+0000 7f91899c6700 1 --2- 172.21.15.129:0/1299815680 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f918406a760 0x7f918406ab40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f91780099c0 tx=0x7f9178009cd0 comp rx=0 tx=0).stop 2024-03-11T20:51:47.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.193+0000 7f91899c6700 1 -- 172.21.15.129:0/1299815680 shutdown_connections 2024-03-11T20:51:47.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.193+0000 7f91899c6700 1 --2- 172.21.15.129:0/1299815680 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9184107170 0x7f918410ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:47.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.193+0000 7f91899c6700 1 --2- 172.21.15.129:0/1299815680 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f918406a760 0x7f918406ab40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:47.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.193+0000 7f91899c6700 1 --2- 172.21.15.129:0/1299815680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f918406b110 0x7f9184106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:47.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.193+0000 7f91899c6700 1 -- 172.21.15.129:0/1299815680 >> 172.21.15.129:0/1299815680 conn(0x7f9184077ba0 msgr2=0x7f9184077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:47.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.193+0000 7f91899c6700 1 -- 172.21.15.129:0/1299815680 shutdown_connections 2024-03-11T20:51:47.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.193+0000 7f91899c6700 1 -- 172.21.15.129:0/1299815680 wait complete. 2024-03-11T20:51:47.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.194+0000 7f91899c6700 1 Processor -- start 2024-03-11T20:51:47.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.194+0000 7f91899c6700 1 -- start start 2024-03-11T20:51:47.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.194+0000 7f91899c6700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f918406a760 0x7f9184101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:47.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.194+0000 7f91899c6700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f918406b110 0x7f91841015d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:47.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.194+0000 7f91899c6700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9184107170 0x7f9184101b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:47.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.194+0000 7f91899c6700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9184111650 con 0x7f9184107170 2024-03-11T20:51:47.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.194+0000 7f91899c6700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91841114d0 con 0x7f918406b110 2024-03-11T20:51:47.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.194+0000 7f91899c6700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91841117d0 con 0x7f918406a760 2024-03-11T20:51:47.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.194+0000 7f9182ffd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f918406a760 0x7f9184101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:47.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.194+0000 7f9182ffd700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f918406a760 0x7f9184101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:60742/0 (socket says 172.21.15.129:60742) 2024-03-11T20:51:47.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.194+0000 7f9182ffd700 1 -- 172.21.15.129:0/1679368187 learned_addr learned my addr 172.21.15.129:0/1679368187 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:51:47.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.194+0000 7f91837fe700 1 --2- 172.21.15.129:0/1679368187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9184107170 0x7f9184101b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:47.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.194+0000 7f91827fc700 1 --2- 172.21.15.129:0/1679368187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f918406b110 0x7f91841015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:47.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.195+0000 7f9182ffd700 1 -- 172.21.15.129:0/1679368187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f918406b110 msgr2=0x7f91841015d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:47.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.195+0000 7f9182ffd700 1 --2- 172.21.15.129:0/1679368187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f918406b110 0x7f91841015d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:47.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.195+0000 7f9182ffd700 1 -- 172.21.15.129:0/1679368187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9184107170 msgr2=0x7f9184101b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:47.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.195+0000 7f9182ffd700 1 --2- 172.21.15.129:0/1679368187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9184107170 0x7f9184101b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:47.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.195+0000 7f9182ffd700 1 -- 172.21.15.129:0/1679368187 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91841a6090 con 0x7f918406a760 2024-03-11T20:51:47.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.195+0000 7f91827fc700 1 --2- 172.21.15.129:0/1679368187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f918406b110 0x7f91841015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T20:51:47.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.195+0000 7f91837fe700 1 --2- 172.21.15.129:0/1679368187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9184107170 0x7f9184101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:51:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.195+0000 7f9182ffd700 1 --2- 172.21.15.129:0/1679368187 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f918406a760 0x7f9184101090 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9178006010 tx=0x7f91780045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.195+0000 7f91889c4700 1 -- 172.21.15.129:0/1679368187 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9178026070 con 0x7f918406a760 2024-03-11T20:51:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.195+0000 7f91899c6700 1 -- 172.21.15.129:0/1679368187 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91841a6320 con 0x7f918406a760 2024-03-11T20:51:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.195+0000 7f91889c4700 1 -- 172.21.15.129:0/1679368187 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9178003e50 con 0x7f918406a760 2024-03-11T20:51:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.195+0000 7f91889c4700 1 -- 172.21.15.129:0/1679368187 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917800fb30 con 0x7f918406a760 2024-03-11T20:51:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.195+0000 7f91899c6700 1 -- 172.21.15.129:0/1679368187 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91841a67d0 con 0x7f918406a760 2024-03-11T20:51:47.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.196+0000 7f91899c6700 1 -- 172.21.15.129:0/1679368187 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918410a340 con 0x7f918406a760 2024-03-11T20:51:47.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.196+0000 7f91889c4700 1 -- 172.21.15.129:0/1679368187 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 16) v1 ==== 91806+0+0 (secure 0 0 0) 0x7f9178029070 con 0x7f918406a760 2024-03-11T20:51:47.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.197+0000 7f91889c4700 1 --2- 172.21.15.129:0/1679368187 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f916c06de20 0x7f916c0702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:47.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.197+0000 7f91889c4700 1 -- 172.21.15.129:0/1679368187 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7207+0+0 (secure 0 0 0) 0x7f917808ea60 con 0x7f918406a760 2024-03-11T20:51:47.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.197+0000 7f91827fc700 1 --2- 172.21.15.129:0/1679368187 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f916c06de20 0x7f916c0702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:47.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.197+0000 7f91827fc700 1 --2- 172.21.15.129:0/1679368187 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f916c06de20 0x7f916c0702e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9184100c60 tx=0x7f9174009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:47.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.202+0000 7f91889c4700 1 -- 172.21.15.129:0/1679368187 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f9178013070 con 0x7f918406a760 2024-03-11T20:51:47.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.396+0000 7f91899c6700 1 -- 172.21.15.129:0/1679368187 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91841025c0 con 0x7f918406a760 2024-03-11T20:51:47.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.398+0000 7f91889c4700 1 -- 172.21.15.129:0/1679368187 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+22809 (secure 0 0 0) 0x7f917805bf70 con 0x7f918406a760 2024-03-11T20:51:47.394 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:51:47.394 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"fsid":"27bbebfa-dfe8-11ee-95c8-87774f69a715","created":"2024-03-11T20:45:35.170893+0000","modified":"2024-03-11T20:51:31.742907+0000","last_up_change":"2024-03-11T20:50:57.449832+0000","last_in_change":"2024-03-11T20:50:39.277288+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T20:48:49.904657+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-11T20:51:09.381905+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-11T20:51:10.107130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-11T20:51:12.032362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-11T20:51:13.974387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T20:51:16.043073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5dbe3ee2-8c30-4242-baf0-a36e1ac0be8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":3618360833},{"type":"v1","addr":"172.21.15.129:6803","nonce":3618360833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":3618360833},{"type":"v1","addr":"172.21.15.129:6805","nonce":3618360833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":3618360833},{"type":"v1","addr":"172.21.15.129:6809","nonce":3618360833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":3618360833},{"type":"v1","addr":"172.21.15.129:6807","nonce":3618360833}]},"public_addr":"172.21.15.129:6803/3618360833","cluster_addr":"172.21.15.129:6805/3618360833","heartbeat_back_addr":"172.21.15.129:6809/3618360833","heartbeat_front_addr":"172.21.15.129:6807/3618360833","state":["exists","up"]},{"osd":1,"uuid":"d5495628-e325-4a58-b682-50a26ef780fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2519234527},{"type":"v1","addr":"172.21.15.129:6811","nonce":2519234527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2519234527},{"type":"v1","addr":"172.21.15.129:6813","nonce":2519234527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":2519234527},{"type":"v1","addr":"172.21.15.129:6817","nonce":2519234527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2519234527},{"type":"v1","addr":"172.21.15.129:6815","nonce":2519234527}]},"public_addr":"172.21.15.129:6811/2519234527","cluster_addr":"172.21.15.129:6813/2519234527","heartbeat_back_addr":"172.21.15.129:6817/2519234527","heartbeat_front_addr":"172.21.15.129:6815/2519234527","state":["exists","up"]},{"osd":2,"uuid":"2b6db120-1132-4369-9f17-777545ad4a07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":1600233087},{"type":"v1","addr":"172.21.15.129:6819","nonce":1600233087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":1600233087},{"type":"v1","addr":"172.21.15.129:6821","nonce":1600233087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1600233087},{"type":"v1","addr":"172.21.15.129:6825","nonce":1600233087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":1600233087},{"type":"v1","addr":"172.21.15.129:6823","nonce":1600233087}]},"public_addr":"172.21.15.129:6819/1600233087","cluster_addr":"172.21.15.129:6821/1600233087","heartbeat_back_addr":"172.21.15.129:6825/1600233087","heartbeat_front_addr":"172.21.15.129:6823/1600233087","state":["exists","up"]},{"osd":3,"uuid":"d9d3db4d-385d-4745-b71f-0796ff551f4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":2265856698},{"type":"v1","addr":"172.21.15.129:6827","nonce":2265856698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":2265856698},{"type":"v1","addr":"172.21.15.129:6829","nonce":2265856698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2265856698},{"type":"v1","addr":"172.21.15.129:6833","nonce":2265856698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":2265856698},{"type":"v1","addr":"172.21.15.129:6831","nonce":2265856698}]},"public_addr":"172.21.15.129:6827/2265856698","cluster_addr":"172.21.15.129:6829/2265856698","heartbeat_back_addr":"172.21.15.129:6833/2265856698","heartbeat_front_addr":"172.21.15.129:6831/2265856698","state":["exists","up"]},{"osd":4,"uuid":"2aa92e2f-5687-4260-985a-3006da3cc803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":3991533346},{"type":"v1","addr":"172.21.15.157:6801","nonce":3991533346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":3991533346},{"type":"v1","addr":"172.21.15.157:6803","nonce":3991533346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":3991533346},{"type":"v1","addr":"172.21.15.157:6807","nonce":3991533346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":3991533346},{"type":"v1","addr":"172.21.15.157:6805","nonce":3991533346}]},"public_addr":"172.21.15.157:6801/3991533346","cluster_addr":"172.21.15.157:6803/3991533346","heartbeat_back_addr":"172.21.15.157:6807/3991533346","heartbeat_front_addr":"172.21.15.157:6805/3991533346","state":["exists","up"]},{"osd":5,"uuid":"dac5fa60-f01b-4323-8596-cc0f3a5030c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":2444017632},{"type":"v1","addr":"172.21.15.157:6809","nonce":2444017632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":2444017632},{"type":"v1","addr":"172.21.15.157:6811","nonce":2444017632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":2444017632},{"type":"v1","addr":"172.21.15.157:6815","nonce":2444017632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":2444017632},{"type":"v1","addr":"172.21.15.157:6813","nonce":2444017632}]},"public_addr":"172.21.15.157:6809/2444017632","cluster_addr":"172.21.15.157:6811/2444017632","heartbeat_back_addr":"172.21.15.157:6815/2444017632","heartbeat_front_addr":"172.21.15.157:6813/2444017632","state":["exists","up"]},{"osd":6,"uuid":"9c489abd-5f6c-4fb9-954a-aabd409fffd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":2710731257},{"type":"v1","addr":"172.21.15.157:6817","nonce":2710731257}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":2710731257},{"type":"v1","addr":"172.21.15.157:6819","nonce":2710731257}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":2710731257},{"type":"v1","addr":"172.21.15.157:6823","nonce":2710731257}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":2710731257},{"type":"v1","addr":"172.21.15.157:6821","nonce":2710731257}]},"public_addr":"172.21.15.157:6817/2710731257","cluster_addr":"172.21.15.157:6819/2710731257","heartbeat_back_addr":"172.21.15.157:6823/2710731257","heartbeat_front_addr":"172.21.15.157:6821/2710731257","state":["exists","up"]},{"osd":7,"uuid":"22bacf1f-35c5-4dee-81a6-92dff2f076c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1786686696},{"type":"v1","addr":"172.21.15.157:6825","nonce":1786686696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1786686696},{"type":"v1","addr":"172.21.15.157:6827","nonce":1786686696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1786686696},{"type":"v1","addr":"172.21.15.157:6831","nonce":1786686696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1786686696},{"type":"v1","addr":"172.21.15.157:6829","nonce":1786686696}]},"public_addr":"172.21.15.157:6825/1786686696","cluster_addr":"172.21.15.157:6827/1786686696","heartbeat_back_addr":"172.21.15.157:6831/1786686696","heartbeat_front_addr":"172.21.15.157:6829/1786686696","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:47:54.931014+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:48:20.791637+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:48:46.661578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:49:13.177044+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:49:37.603176+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:50:03.465298+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:50:29.681381+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T20:50:55.559486+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:0/2734363055":"2024-03-12T20:46:29.573760+0000","172.21.15.129:0/1185914521":"2024-03-12T20:46:29.573760+0000","172.21.15.129:6800/2213467472":"2024-03-12T20:46:02.671885+0000","172.21.15.129:0/1229097270":"2024-03-12T20:46:02.671885+0000","172.21.15.129:6801/521270824":"2024-03-12T20:46:29.573760+0000","172.21.15.129:6801/2213467472":"2024-03-12T20:46:02.671885+0000","172.21.15.129:0/160155772":"2024-03-12T20:46:29.573760+0000","172.21.15.129:0/4048407456":"2024-03-12T20:46:02.671885+0000","172.21.15.129:0/1059875431":"2024-03-12T20:46:02.671885+0000","172.21.15.129:6800/521270824":"2024-03-12T20:46:29.573760+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T20:51:47.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.400+0000 7f91899c6700 1 -- 172.21.15.129:0/1679368187 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f916c06de20 msgr2=0x7f916c0702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:47.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.400+0000 7f91899c6700 1 --2- 172.21.15.129:0/1679368187 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f916c06de20 0x7f916c0702e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9184100c60 tx=0x7f9174009260 comp rx=0 tx=0).stop 2024-03-11T20:51:47.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.400+0000 7f91899c6700 1 -- 172.21.15.129:0/1679368187 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f918406a760 msgr2=0x7f9184101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:47.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.400+0000 7f91899c6700 1 --2- 172.21.15.129:0/1679368187 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f918406a760 0x7f9184101090 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9178006010 tx=0x7f91780045e0 comp rx=0 tx=0).stop 2024-03-11T20:51:47.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.400+0000 7f91899c6700 1 -- 172.21.15.129:0/1679368187 shutdown_connections 2024-03-11T20:51:47.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.400+0000 7f91899c6700 1 --2- 172.21.15.129:0/1679368187 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f916c06de20 0x7f916c0702e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:47.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.400+0000 7f91899c6700 1 --2- 172.21.15.129:0/1679368187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9184107170 0x7f9184101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:47.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.400+0000 7f91899c6700 1 --2- 172.21.15.129:0/1679368187 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f918406a760 0x7f9184101090 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:47.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.400+0000 7f91899c6700 1 --2- 172.21.15.129:0/1679368187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f918406b110 0x7f91841015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:47.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.400+0000 7f91899c6700 1 -- 172.21.15.129:0/1679368187 >> 172.21.15.129:0/1679368187 conn(0x7f9184077ba0 msgr2=0x7f91840ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:47.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.400+0000 7f91899c6700 1 -- 172.21.15.129:0/1679368187 shutdown_connections 2024-03-11T20:51:47.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:47.400+0000 7f91899c6700 1 -- 172.21.15.129:0/1679368187 wait complete. 2024-03-11T20:51:47.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1679368187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:51:47.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1679368187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:51:47.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1679368187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T20:51:48.157 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T20:51:48.158 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T20:51:48.158 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T20:51:48.159 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-11T20:51:48.159 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-11T20:51:48.160 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-11T20:51:48.160 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-11T20:51:48.160 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-11T20:51:48.460 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:48 smithi129 ceph-mon[42066]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:51:48.687 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:51:48.687 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:51:48.688 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:51:48.688 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:51:48.690 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:51:48.693 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:51:48.694 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:51:48.703 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:51:48.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:48 smithi157 ceph-mon[33359]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:51:48.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:48 smithi129 ceph-mon[30184]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:51:50.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:50 smithi129 ceph-mon[30184]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:51:50.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:50 smithi129 ceph-mon[42066]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:51:50.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:50 smithi157 ceph-mon[33359]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:51:52.222 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 systemd[1]: Starting Ceph prometheus.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T20:51:52.474 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 podman[64272]: 2024-03-11 20:51:52.227098591 +0000 UTC m=+0.100508354 container create 3f2859b29719b4748ac770b38078c33445c856e2060b658629b053137882ab0e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T20:51:52.474 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 podman[64272]: 2024-03-11 20:51:52.146623254 +0000 UTC m=+0.020032981 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-11T20:51:52.474 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 podman[64272]: 2024-03-11 20:51:52.445787047 +0000 UTC m=+0.319196769 container init 3f2859b29719b4748ac770b38078c33445c856e2060b658629b053137882ab0e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T20:51:52.474 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 podman[64272]: 2024-03-11 20:51:52.451170897 +0000 UTC m=+0.324580622 container start 3f2859b29719b4748ac770b38078c33445c856e2060b658629b053137882ab0e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T20:51:52.474 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.475Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T20:51:52.475 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.475Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T20:51:52.475 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.475Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi157 (none))" 2024-03-11T20:51:52.475 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.475Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T20:51:52.475 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.475Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T20:51:52.475 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.477Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T20:51:52.475 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.477Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T20:51:52.475 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.478Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T20:51:52.475 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.478Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T20:51:52.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:52 smithi129 ceph-mon[42066]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:51:52.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:52 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:52.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:52 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:52.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:52 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:52.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:52 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T20:51:52.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:52 smithi129 ceph-mon[30184]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:51:52.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:52 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:52.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:52 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:52.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:52 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:52.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:52 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T20:51:52.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-mon[33359]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:51:52.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:52.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:52.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' 2024-03-11T20:51:52.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T20:51:52.736 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.480Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T20:51:52.736 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.480Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.069µs 2024-03-11T20:51:52.737 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.480Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T20:51:52.737 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.481Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-11T20:51:52.737 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.481Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.859µs wal_replay_duration=221.22µs wbl_replay_duration=87ns total_replay_duration=261.013µs 2024-03-11T20:51:52.737 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.481Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T20:51:52.737 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.481Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T20:51:52.737 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.481Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T20:51:52.737 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.502Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.433368ms db_storage=709ns remote_storage=1.128µs web_handler=282ns query_engine=474ns scrape=430.235µs scrape_sd=75.593µs notify=675ns notify_sd=1.345µs rules=19.678648ms tracing=6.461µs 2024-03-11T20:51:52.737 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.502Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T20:51:52.737 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:51:52.502Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T20:51:52.737 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 bash[64272]: 3f2859b29719b4748ac770b38078c33445c856e2060b658629b053137882ab0e 2024-03-11T20:51:52.737 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:51:52 smithi157 systemd[1]: Started Ceph prometheus.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T20:51:53.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:51:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:51:53.728 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:53 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ignoring --setuser ceph since I am not root 2024-03-11T20:51:53.728 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:53 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ignoring --setgroup ceph since I am not root 2024-03-11T20:51:53.985 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: ignoring --setuser ceph since I am not root 2024-03-11T20:51:53.986 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: ignoring --setgroup ceph since I am not root 2024-03-11T20:51:53.986 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:53 smithi157 ceph-mgr[35480]: -- 172.21.15.157:0/3667955105 <== mon.1 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555a980025a0 con 0x555a97fd6800 2024-03-11T20:51:54.315 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:54 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:54.220+0000 7f0e7985a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T20:51:54.315 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:54 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:54.318+0000 7f0e7985a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T20:51:54.485 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:54.187+0000 7fc24708f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T20:51:54.486 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:54.280+0000 7fc24708f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T20:51:54.870 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:54 smithi129 ceph-mon[30184]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T20:51:54.870 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:51:54 smithi129 ceph-mon[30184]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-11T20:51:54.870 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:54 smithi129 ceph-mon[42066]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T20:51:54.871 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:51:54 smithi129 ceph-mon[42066]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-11T20:51:54.911 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:54 smithi157 ceph-mon[33359]: from='mgr.14152 172.21.15.129:0/261413564' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T20:51:54.911 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:51:54 smithi157 ceph-mon[33359]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-11T20:51:55.190 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:54 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:54.987+0000 7f0e7985a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T20:51:55.190 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:55 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:55.080+0000 7f0e7985a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T20:51:55.235 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:54.915+0000 7fc24708f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T20:51:55.235 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:55.008+0000 7fc24708f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T20:51:55.235 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:55.152+0000 7fc24708f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T20:51:55.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.300+0000 7f932445d700 1 -- 172.21.15.129:0/2795483858 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f931c101e60 msgr2=0x7f931c1022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:55.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.300+0000 7f932445d700 1 --2- 172.21.15.129:0/2795483858 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f931c101e60 0x7f931c1022e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9318009990 tx=0x7f9318009ca0 comp rx=0 tx=0).stop 2024-03-11T20:51:55.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.300+0000 7f932445d700 1 -- 172.21.15.129:0/2795483858 shutdown_connections 2024-03-11T20:51:55.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.300+0000 7f932445d700 1 --2- 172.21.15.129:0/2795483858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f931c102820 0x7f931c10ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:55.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.300+0000 7f932445d700 1 --2- 172.21.15.129:0/2795483858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f931c107e70 0x7f931c108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:55.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.300+0000 7f932445d700 1 --2- 172.21.15.129:0/2795483858 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f931c101e60 0x7f931c1022e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:55.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.300+0000 7f932445d700 1 -- 172.21.15.129:0/2795483858 >> 172.21.15.129:0/2795483858 conn(0x7f931c078f40 msgr2=0x7f931c079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:55.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.300+0000 7f932445d700 1 -- 172.21.15.129:0/2795483858 shutdown_connections 2024-03-11T20:51:55.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.300+0000 7f932445d700 1 -- 172.21.15.129:0/2795483858 wait complete. 2024-03-11T20:51:55.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.301+0000 7f932445d700 1 Processor -- start 2024-03-11T20:51:55.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.301+0000 7f932445d700 1 -- start start 2024-03-11T20:51:55.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.301+0000 7f932445d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f931c101e60 0x7f931c19d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:55.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.301+0000 7f932445d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f931c102820 0x7f931c19dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:55.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.301+0000 7f932445d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f931c107e70 0x7f931c197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:55.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.301+0000 7f932445d700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c111640 con 0x7f931c107e70 2024-03-11T20:51:55.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.301+0000 7f932445d700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c1114c0 con 0x7f931c101e60 2024-03-11T20:51:55.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.301+0000 7f932445d700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931c1117c0 con 0x7f931c102820 2024-03-11T20:51:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.301+0000 7f93229fa700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f931c107e70 0x7f931c197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.301+0000 7f93219f8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f931c102820 0x7f931c19dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.301+0000 7f93229fa700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f931c107e70 0x7f931c197880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48654/0 (socket says 172.21.15.129:48654) 2024-03-11T20:51:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.301+0000 7f93229fa700 1 -- 172.21.15.129:0/4135768460 learned_addr learned my addr 172.21.15.129:0/4135768460 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:51:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.301+0000 7f93221f9700 1 --2- 172.21.15.129:0/4135768460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f931c101e60 0x7f931c19d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.301+0000 7f93229fa700 1 -- 172.21.15.129:0/4135768460 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f931c102820 msgr2=0x7f931c19dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.302+0000 7f93229fa700 1 --2- 172.21.15.129:0/4135768460 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f931c102820 0x7f931c19dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.302+0000 7f93229fa700 1 -- 172.21.15.129:0/4135768460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f931c101e60 msgr2=0x7f931c19d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.302+0000 7f93229fa700 1 --2- 172.21.15.129:0/4135768460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f931c101e60 0x7f931c19d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.302+0000 7f93229fa700 1 -- 172.21.15.129:0/4135768460 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c197fe0 con 0x7f931c107e70 2024-03-11T20:51:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.302+0000 7f93219f8700 1 --2- 172.21.15.129:0/4135768460 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f931c102820 0x7f931c19dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:51:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.302+0000 7f93221f9700 1 --2- 172.21.15.129:0/4135768460 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f931c101e60 0x7f931c19d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:51:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.302+0000 7f93229fa700 1 --2- 172.21.15.129:0/4135768460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f931c107e70 0x7f931c197880 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f931400c0f0 tx=0x7f931400c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:55.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.302+0000 7f93137fe700 1 -- 172.21.15.129:0/4135768460 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9314018020 con 0x7f931c107e70 2024-03-11T20:51:55.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.302+0000 7f93137fe700 1 -- 172.21.15.129:0/4135768460 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f931400ca00 con 0x7f931c107e70 2024-03-11T20:51:55.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.302+0000 7f932445d700 1 -- 172.21.15.129:0/4135768460 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931c1982d0 con 0x7f931c107e70 2024-03-11T20:51:55.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.302+0000 7f93137fe700 1 -- 172.21.15.129:0/4135768460 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9314004530 con 0x7f931c107e70 2024-03-11T20:51:55.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.302+0000 7f932445d700 1 -- 172.21.15.129:0/4135768460 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f931c1a4590 con 0x7f931c107e70 2024-03-11T20:51:55.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.303+0000 7f932445d700 1 -- 172.21.15.129:0/4135768460 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f931c0624f0 con 0x7f931c107e70 2024-03-11T20:51:55.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.303+0000 7f93137fe700 1 -- 172.21.15.129:0/4135768460 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) v1 ==== 91820+0+0 (secure 0 0 0) 0x7f9314012050 con 0x7f931c107e70 2024-03-11T20:51:55.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.304+0000 7f93137fe700 1 --2- 172.21.15.129:0/4135768460 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f930806de70 0x7f9308070330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:55.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.304+0000 7f93221f9700 1 -- 172.21.15.129:0/4135768460 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f930806de70 msgr2=0x7f9308070330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2866184010 2024-03-11T20:51:55.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.304+0000 7f93137fe700 1 -- 172.21.15.129:0/4135768460 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7207+0+0 (secure 0 0 0) 0x7f931408dd70 con 0x7f931c107e70 2024-03-11T20:51:55.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.304+0000 7f93221f9700 1 --2- 172.21.15.129:0/4135768460 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f930806de70 0x7f9308070330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T20:51:55.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.304+0000 7f93137fe700 1 --2- 172.21.15.129:0/4135768460 >> [v2:172.21.15.157:6800/3991533346,v1:172.21.15.157:6801/3991533346] conn(0x7f9308077a30 0x7f9308079ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:55.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.304+0000 7f93137fe700 1 -- 172.21.15.129:0/4135768460 --> [v2:172.21.15.157:6800/3991533346,v1:172.21.15.157:6801/3991533346] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9314016ec0 con 0x7f9308077a30 2024-03-11T20:51:55.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.304+0000 7f93137fe700 1 -- 172.21.15.129:0/4135768460 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93140904c0 con 0x7f931c107e70 2024-03-11T20:51:55.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.304+0000 7f93219f8700 1 --2- 172.21.15.129:0/4135768460 >> [v2:172.21.15.157:6800/3991533346,v1:172.21.15.157:6801/3991533346] conn(0x7f9308077a30 0x7f9308079ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:55.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.305+0000 7f93219f8700 1 --2- 172.21.15.129:0/4135768460 >> [v2:172.21.15.157:6800/3991533346,v1:172.21.15.157:6801/3991533346] conn(0x7f9308077a30 0x7f9308079ed0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:55.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.305+0000 7f93137fe700 1 -- 172.21.15.129:0/4135768460 <== osd.4 v2:172.21.15.157:6800/3991533346 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9314016ec0 con 0x7f9308077a30 2024-03-11T20:51:55.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.321+0000 7f932445d700 1 -- 172.21.15.129:0/4135768460 --> [v2:172.21.15.157:6800/3991533346,v1:172.21.15.157:6801/3991533346] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f931c0628f0 con 0x7f9308077a30 2024-03-11T20:51:55.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.322+0000 7f93137fe700 1 -- 172.21.15.129:0/4135768460 <== osd.4 v2:172.21.15.157:6800/3991533346 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f931c0628f0 con 0x7f9308077a30 2024-03-11T20:51:55.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.322+0000 7f932445d700 1 -- 172.21.15.129:0/4135768460 >> [v2:172.21.15.157:6800/3991533346,v1:172.21.15.157:6801/3991533346] conn(0x7f9308077a30 msgr2=0x7f9308079ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:55.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.322+0000 7f932445d700 1 --2- 172.21.15.129:0/4135768460 >> [v2:172.21.15.157:6800/3991533346,v1:172.21.15.157:6801/3991533346] conn(0x7f9308077a30 0x7f9308079ed0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:55.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.322+0000 7f932445d700 1 -- 172.21.15.129:0/4135768460 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f930806de70 msgr2=0x7f9308070330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T20:51:55.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.322+0000 7f932445d700 1 --2- 172.21.15.129:0/4135768460 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f930806de70 0x7f9308070330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:55.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.322+0000 7f932445d700 1 -- 172.21.15.129:0/4135768460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f931c107e70 msgr2=0x7f931c197880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:55.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.322+0000 7f932445d700 1 --2- 172.21.15.129:0/4135768460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f931c107e70 0x7f931c197880 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f931400c0f0 tx=0x7f931400c400 comp rx=0 tx=0).stop 2024-03-11T20:51:55.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.322+0000 7f93229fa700 1 -- 172.21.15.129:0/4135768460 reap_dead start 2024-03-11T20:51:55.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.323+0000 7f932445d700 1 -- 172.21.15.129:0/4135768460 shutdown_connections 2024-03-11T20:51:55.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.323+0000 7f932445d700 1 -- 172.21.15.129:0/4135768460 >> 172.21.15.129:0/4135768460 conn(0x7f931c078f40 msgr2=0x7f931c100b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:55.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.323+0000 7f932445d700 1 -- 172.21.15.129:0/4135768460 shutdown_connections 2024-03-11T20:51:55.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:55.323+0000 7f932445d700 1 -- 172.21.15.129:0/4135768460 wait complete. 2024-03-11T20:51:55.478 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:55 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:55.224+0000 7f0e7985a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T20:51:55.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:55 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:55.326+0000 7f0e7985a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T20:51:55.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:55 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:55.416+0000 7f0e7985a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T20:51:55.589 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:55.247+0000 7fc24708f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T20:51:55.589 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:55.331+0000 7fc24708f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T20:51:55.589 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:55.593+0000 7fc24708f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T20:51:55.884 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:55.704+0000 7fc24708f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T20:51:55.885 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:55.793+0000 7fc24708f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T20:51:55.973 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:55 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:55.684+0000 7f0e7985a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T20:51:55.974 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:55 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:55.796+0000 7f0e7985a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T20:51:55.974 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:55 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:55.887+0000 7f0e7985a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T20:51:56.228 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:55 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:55.986+0000 7f0e7985a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T20:51:56.228 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:56.162+0000 7f0e7985a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T20:51:56.235 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:55.888+0000 7fc24708f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T20:51:56.235 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:56.065+0000 7fc24708f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T20:51:57.276 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:57 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:57.145+0000 7f0e7985a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T20:51:57.484 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:57 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:57.026+0000 7fc24708f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T20:51:57.484 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:57 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:57.156+0000 7fc24708f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T20:51:57.600 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:57 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:57.279+0000 7f0e7985a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T20:51:57.735 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:57 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:57.488+0000 7fc24708f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T20:51:57.735 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:57 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:57.578+0000 7fc24708f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T20:51:57.886 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:57 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:57.621+0000 7f0e7985a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T20:51:57.886 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:57 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:57.720+0000 7f0e7985a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T20:51:58.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.132+0000 7f6ee4e79700 1 -- 172.21.15.129:0/2457129115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ee0100fc0 msgr2=0x7f6ee0101440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:58.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.132+0000 7f6ee4e79700 1 --2- 172.21.15.129:0/2457129115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ee0100fc0 0x7f6ee0101440 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00099c0 tx=0x7f6ed0009cd0 comp rx=0 tx=0).stop 2024-03-11T20:51:58.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.132+0000 7f6ee4e79700 1 -- 172.21.15.129:0/2457129115 shutdown_connections 2024-03-11T20:51:58.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.132+0000 7f6ee4e79700 1 --2- 172.21.15.129:0/2457129115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ee0101980 0x7f6ee010dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:58.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.132+0000 7f6ee4e79700 1 --2- 172.21.15.129:0/2457129115 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6ee0106e10 0x7f6ee01071f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:58.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.132+0000 7f6ee4e79700 1 --2- 172.21.15.129:0/2457129115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ee0100fc0 0x7f6ee0101440 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:58.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.132+0000 7f6ee4e79700 1 -- 172.21.15.129:0/2457129115 >> 172.21.15.129:0/2457129115 conn(0x7f6ee0077ba0 msgr2=0x7f6ee0077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:58.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.132+0000 7f6ee4e79700 1 -- 172.21.15.129:0/2457129115 shutdown_connections 2024-03-11T20:51:58.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.132+0000 7f6ee4e79700 1 -- 172.21.15.129:0/2457129115 wait complete. 2024-03-11T20:51:58.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.133+0000 7f6ee4e79700 1 Processor -- start 2024-03-11T20:51:58.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.133+0000 7f6ee4e79700 1 -- start start 2024-03-11T20:51:58.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.133+0000 7f6ee4e79700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ee0100fc0 0x7f6ee019c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:58.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.133+0000 7f6ee4e79700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ee0101980 0x7f6ee019cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:58.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.133+0000 7f6ee4e79700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6ee0106e10 0x7f6ee0196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:58.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.133+0000 7f6ee4e79700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee0110550 con 0x7f6ee0100fc0 2024-03-11T20:51:58.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.133+0000 7f6ee4e79700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee01103d0 con 0x7f6ee0101980 2024-03-11T20:51:58.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.133+0000 7f6ee4e79700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee01106d0 con 0x7f6ee0106e10 2024-03-11T20:51:58.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.134+0000 7f6ede59c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ee0100fc0 0x7f6ee019c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:58.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.134+0000 7f6ede59c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ee0100fc0 0x7f6ee019c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50734/0 (socket says 172.21.15.129:50734) 2024-03-11T20:51:58.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.134+0000 7f6ede59c700 1 -- 172.21.15.129:0/3274239758 learned_addr learned my addr 172.21.15.129:0/3274239758 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:51:58.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.134+0000 7f6eddd9b700 1 --2- 172.21.15.129:0/3274239758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ee0101980 0x7f6ee019cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:58.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.134+0000 7f6eded9d700 1 --2- 172.21.15.129:0/3274239758 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6ee0106e10 0x7f6ee0196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:58.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.134+0000 7f6eddd9b700 1 -- 172.21.15.129:0/3274239758 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6ee0106e10 msgr2=0x7f6ee0196790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:58.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.134+0000 7f6eddd9b700 1 --2- 172.21.15.129:0/3274239758 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f6ee0106e10 0x7f6ee0196790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:58.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.134+0000 7f6eddd9b700 1 -- 172.21.15.129:0/3274239758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ee0100fc0 msgr2=0x7f6ee019c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:58.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.134+0000 7f6eddd9b700 1 --2- 172.21.15.129:0/3274239758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ee0100fc0 0x7f6ee019c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:58.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.134+0000 7f6eddd9b700 1 -- 172.21.15.129:0/3274239758 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee0196f70 con 0x7f6ee0101980 2024-03-11T20:51:58.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.134+0000 7f6eddd9b700 1 --2- 172.21.15.129:0/3274239758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ee0101980 0x7f6ee019cb50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000b580 tx=0x7f6ed0005280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:58.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.134+0000 7f6ecf7fe700 1 -- 172.21.15.129:0/3274239758 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0026070 con 0x7f6ee0101980 2024-03-11T20:51:58.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.134+0000 7f6ecf7fe700 1 -- 172.21.15.129:0/3274239758 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f6ed00056c0 con 0x7f6ee0101980 2024-03-11T20:51:58.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.134+0000 7f6ecf7fe700 1 -- 172.21.15.129:0/3274239758 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed000f7e0 con 0x7f6ee0101980 2024-03-11T20:51:58.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.134+0000 7f6ee4e79700 1 -- 172.21.15.129:0/3274239758 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee0197200 con 0x7f6ee0101980 2024-03-11T20:51:58.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.135+0000 7f6ee4e79700 1 -- 172.21.15.129:0/3274239758 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee006a480 con 0x7f6ee0101980 2024-03-11T20:51:58.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.135+0000 7f6ee4e79700 1 -- 172.21.15.129:0/3274239758 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6ee006a9b0 con 0x7f6ee0101980 2024-03-11T20:51:58.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.136+0000 7f6ecf7fe700 1 -- 172.21.15.129:0/3274239758 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 17) v1 ==== 91820+0+0 (secure 0 0 0) 0x7f6ed0029070 con 0x7f6ee0101980 2024-03-11T20:51:58.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.137+0000 7f6ecf7fe700 1 --2- 172.21.15.129:0/3274239758 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6ec806de70 0x7f6ec8070330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:58.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.137+0000 7f6ede59c700 1 -- 172.21.15.129:0/3274239758 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6ec806de70 msgr2=0x7f6ec8070330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2866184010 2024-03-11T20:51:58.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.137+0000 7f6ecf7fe700 1 -- 172.21.15.129:0/3274239758 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7207+0+0 (secure 0 0 0) 0x7f6ed008e8f0 con 0x7f6ee0101980 2024-03-11T20:51:58.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.137+0000 7f6ede59c700 1 --2- 172.21.15.129:0/3274239758 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6ec806de70 0x7f6ec8070330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T20:51:58.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.137+0000 7f6ecf7fe700 1 --2- 172.21.15.129:0/3274239758 >> [v2:172.21.15.157:6816/2710731257,v1:172.21.15.157:6817/2710731257] conn(0x7f6ec8077ab0 0x7f6ec8079f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:58.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.137+0000 7f6ecf7fe700 1 -- 172.21.15.129:0/3274239758 --> [v2:172.21.15.157:6816/2710731257,v1:172.21.15.157:6817/2710731257] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ed0010e50 con 0x7f6ec8077ab0 2024-03-11T20:51:58.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.137+0000 7f6ecf7fe700 1 -- 172.21.15.129:0/3274239758 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ed008edd0 con 0x7f6ee0101980 2024-03-11T20:51:58.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.137+0000 7f6eded9d700 1 --2- 172.21.15.129:0/3274239758 >> [v2:172.21.15.157:6816/2710731257,v1:172.21.15.157:6817/2710731257] conn(0x7f6ec8077ab0 0x7f6ec8079f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:58.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.137+0000 7f6eded9d700 1 --2- 172.21.15.129:0/3274239758 >> [v2:172.21.15.157:6816/2710731257,v1:172.21.15.157:6817/2710731257] conn(0x7f6ec8077ab0 0x7f6ec8079f50 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:58.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.138+0000 7f6ecf7fe700 1 -- 172.21.15.129:0/3274239758 <== osd.6 v2:172.21.15.157:6816/2710731257 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6ed0010e50 con 0x7f6ec8077ab0 2024-03-11T20:51:58.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.155+0000 7f6ee4e79700 1 -- 172.21.15.129:0/3274239758 --> [v2:172.21.15.157:6816/2710731257,v1:172.21.15.157:6817/2710731257] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6ee006ae00 con 0x7f6ec8077ab0 2024-03-11T20:51:58.152 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:57 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:57.927+0000 7f0e7985a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:51:58.152 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:58 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:58.119+0000 7f0e7985a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T20:51:58.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.155+0000 7f6ecf7fe700 1 -- 172.21.15.129:0/3274239758 <== osd.6 v2:172.21.15.157:6816/2710731257 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6ee006ae00 con 0x7f6ec8077ab0 2024-03-11T20:51:58.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.155+0000 7f6ee4e79700 1 -- 172.21.15.129:0/3274239758 >> [v2:172.21.15.157:6816/2710731257,v1:172.21.15.157:6817/2710731257] conn(0x7f6ec8077ab0 msgr2=0x7f6ec8079f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:58.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.155+0000 7f6ee4e79700 1 --2- 172.21.15.129:0/3274239758 >> [v2:172.21.15.157:6816/2710731257,v1:172.21.15.157:6817/2710731257] conn(0x7f6ec8077ab0 0x7f6ec8079f50 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:58.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.155+0000 7f6ee4e79700 1 -- 172.21.15.129:0/3274239758 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6ec806de70 msgr2=0x7f6ec8070330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T20:51:58.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.155+0000 7f6ee4e79700 1 --2- 172.21.15.129:0/3274239758 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f6ec806de70 0x7f6ec8070330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:58.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.155+0000 7f6ee4e79700 1 -- 172.21.15.129:0/3274239758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ee0101980 msgr2=0x7f6ee019cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:58.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.155+0000 7f6ee4e79700 1 --2- 172.21.15.129:0/3274239758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ee0101980 0x7f6ee019cb50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000b580 tx=0x7f6ed0005280 comp rx=0 tx=0).stop 2024-03-11T20:51:58.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.155+0000 7f6eded9d700 1 -- 172.21.15.129:0/3274239758 reap_dead start 2024-03-11T20:51:58.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.155+0000 7f6ee4e79700 1 -- 172.21.15.129:0/3274239758 shutdown_connections 2024-03-11T20:51:58.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.155+0000 7f6ee4e79700 1 -- 172.21.15.129:0/3274239758 >> 172.21.15.129:0/3274239758 conn(0x7f6ee0077ba0 msgr2=0x7f6ee00ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:58.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.156+0000 7f6ee4e79700 1 -- 172.21.15.129:0/3274239758 shutdown_connections 2024-03-11T20:51:58.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:58.156+0000 7f6ee4e79700 1 -- 172.21.15.129:0/3274239758 wait complete. 2024-03-11T20:51:58.235 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:57 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:57.778+0000 7fc24708f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:51:58.235 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:57 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:57.964+0000 7fc24708f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T20:51:58.508 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:58 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:58.251+0000 7fc24708f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:51:58.687 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:58 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:58.419+0000 7f0e7985a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T20:51:58.954 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:58 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:58.690+0000 7f0e7985a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T20:51:58.954 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:58 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:58.930+0000 7f0e7985a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T20:51:58.985 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:58 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:58.512+0000 7fc24708f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T20:51:58.985 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:58 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:58.751+0000 7fc24708f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T20:51:59.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.061+0000 7f3ea6294700 1 -- 172.21.15.129:0/375690278 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3ea0100ce0 msgr2=0x7f3ea0101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:59.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.061+0000 7f3ea6294700 1 --2- 172.21.15.129:0/375690278 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3ea0100ce0 0x7f3ea0101140 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3e90009990 tx=0x7f3e90009ca0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.061+0000 7f3ea6294700 1 -- 172.21.15.129:0/375690278 shutdown_connections 2024-03-11T20:51:59.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.061+0000 7f3ea6294700 1 --2- 172.21.15.129:0/375690278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ea0101680 0x7f3ea010db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.061+0000 7f3ea6294700 1 --2- 172.21.15.129:0/375690278 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ea0106d20 0x7f3ea0107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.061+0000 7f3ea6294700 1 --2- 172.21.15.129:0/375690278 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3ea0100ce0 0x7f3ea0101140 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.061+0000 7f3ea6294700 1 -- 172.21.15.129:0/375690278 >> 172.21.15.129:0/375690278 conn(0x7f3ea0077c10 msgr2=0x7f3ea0078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:59.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.061+0000 7f3ea6294700 1 -- 172.21.15.129:0/375690278 shutdown_connections 2024-03-11T20:51:59.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.061+0000 7f3ea6294700 1 -- 172.21.15.129:0/375690278 wait complete. 2024-03-11T20:51:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3ea6294700 1 Processor -- start 2024-03-11T20:51:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3ea6294700 1 -- start start 2024-03-11T20:51:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3ea6294700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ea0100ce0 0x7f3ea0198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3ea6294700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ea0101680 0x7f3ea0198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3ea6294700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3ea0106d20 0x7f3ea01a39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3ea6294700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea01104f0 con 0x7f3ea0100ce0 2024-03-11T20:51:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3ea6294700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea0110370 con 0x7f3ea0101680 2024-03-11T20:51:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3ea6294700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea0110670 con 0x7f3ea0106d20 2024-03-11T20:51:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3e9ffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ea0100ce0 0x7f3ea0198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:59.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3e9ffff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ea0100ce0 0x7f3ea0198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50754/0 (socket says 172.21.15.129:50754) 2024-03-11T20:51:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3e9ffff700 1 -- 172.21.15.129:0/1569515087 learned_addr learned my addr 172.21.15.129:0/1569515087 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:51:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3ea4831700 1 --2- 172.21.15.129:0/1569515087 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3ea0106d20 0x7f3ea01a39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3e9f7fe700 1 --2- 172.21.15.129:0/1569515087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ea0101680 0x7f3ea0198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3e9ffff700 1 -- 172.21.15.129:0/1569515087 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3ea0106d20 msgr2=0x7f3ea01a39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3e9ffff700 1 --2- 172.21.15.129:0/1569515087 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3ea0106d20 0x7f3ea01a39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3e9ffff700 1 -- 172.21.15.129:0/1569515087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ea0101680 msgr2=0x7f3ea0198e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.062+0000 7f3e9ffff700 1 --2- 172.21.15.129:0/1569515087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ea0101680 0x7f3ea0198e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.063+0000 7f3e9ffff700 1 -- 172.21.15.129:0/1569515087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea01a3f10 con 0x7f3ea0100ce0 2024-03-11T20:51:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.063+0000 7f3ea4831700 1 --2- 172.21.15.129:0/1569515087 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3ea0106d20 0x7f3ea01a39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:51:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.063+0000 7f3e9f7fe700 1 --2- 172.21.15.129:0/1569515087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ea0101680 0x7f3ea0198e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:51:59.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.063+0000 7f3e9ffff700 1 --2- 172.21.15.129:0/1569515087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ea0100ce0 0x7f3ea0198950 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3e9400e9c0 tx=0x7f3e9400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.063+0000 7f3e9d7fa700 1 -- 172.21.15.129:0/1569515087 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9400cbd0 con 0x7f3ea0100ce0 2024-03-11T20:51:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.063+0000 7f3ea6294700 1 -- 172.21.15.129:0/1569515087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea01a4140 con 0x7f3ea0100ce0 2024-03-11T20:51:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.063+0000 7f3e9d7fa700 1 -- 172.21.15.129:0/1569515087 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3e9400ee50 con 0x7f3ea0100ce0 2024-03-11T20:51:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.063+0000 7f3e9d7fa700 1 -- 172.21.15.129:0/1569515087 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e94005470 con 0x7f3ea0100ce0 2024-03-11T20:51:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.063+0000 7f3ea6294700 1 -- 172.21.15.129:0/1569515087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea01a4650 con 0x7f3ea0100ce0 2024-03-11T20:51:59.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.064+0000 7f3ea6294700 1 -- 172.21.15.129:0/1569515087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3ea00624b0 con 0x7f3ea0100ce0 2024-03-11T20:51:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.065+0000 7f3e9d7fa700 1 -- 172.21.15.129:0/1569515087 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 17) v1 ==== 91820+0+0 (secure 0 0 0) 0x7f3e94004750 con 0x7f3ea0100ce0 2024-03-11T20:51:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.065+0000 7f3e9d7fa700 1 --2- 172.21.15.129:0/1569515087 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f3e8806de70 0x7f3e88070330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.065+0000 7f3e9f7fe700 1 -- 172.21.15.129:0/1569515087 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f3e8806de70 msgr2=0x7f3e88070330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2866184010 2024-03-11T20:51:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.065+0000 7f3e9d7fa700 1 -- 172.21.15.129:0/1569515087 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7207+0+0 (secure 0 0 0) 0x7f3e9408e790 con 0x7f3ea0100ce0 2024-03-11T20:51:59.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.065+0000 7f3e9f7fe700 1 --2- 172.21.15.129:0/1569515087 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f3e8806de70 0x7f3e88070330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T20:51:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.065+0000 7f3e9d7fa700 1 --2- 172.21.15.129:0/1569515087 >> [v2:172.21.15.157:6808/2444017632,v1:172.21.15.157:6809/2444017632] conn(0x7f3e88077a30 0x7f3e88079ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.065+0000 7f3e9d7fa700 1 -- 172.21.15.129:0/1569515087 --> [v2:172.21.15.157:6808/2444017632,v1:172.21.15.157:6809/2444017632] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e94012ec0 con 0x7f3e88077a30 2024-03-11T20:51:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.065+0000 7f3e9d7fa700 1 -- 172.21.15.129:0/1569515087 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e9408ec70 con 0x7f3ea0100ce0 2024-03-11T20:51:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.066+0000 7f3ea4831700 1 --2- 172.21.15.129:0/1569515087 >> [v2:172.21.15.157:6808/2444017632,v1:172.21.15.157:6809/2444017632] conn(0x7f3e88077a30 0x7f3e88079ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.066+0000 7f3ea4831700 1 --2- 172.21.15.129:0/1569515087 >> [v2:172.21.15.157:6808/2444017632,v1:172.21.15.157:6809/2444017632] conn(0x7f3e88077a30 0x7f3e88079ed0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.067+0000 7f3e9d7fa700 1 -- 172.21.15.129:0/1569515087 <== osd.5 v2:172.21.15.157:6808/2444017632 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3e94012ec0 con 0x7f3e88077a30 2024-03-11T20:51:59.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.083+0000 7f3ea6294700 1 -- 172.21.15.129:0/1569515087 --> [v2:172.21.15.157:6808/2444017632,v1:172.21.15.157:6809/2444017632] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3ea00628b0 con 0x7f3e88077a30 2024-03-11T20:51:59.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.084+0000 7f3e9d7fa700 1 -- 172.21.15.129:0/1569515087 <== osd.5 v2:172.21.15.157:6808/2444017632 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f3ea00628b0 con 0x7f3e88077a30 2024-03-11T20:51:59.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.084+0000 7f3ea6294700 1 -- 172.21.15.129:0/1569515087 >> [v2:172.21.15.157:6808/2444017632,v1:172.21.15.157:6809/2444017632] conn(0x7f3e88077a30 msgr2=0x7f3e88079ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:59.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.084+0000 7f3ea6294700 1 --2- 172.21.15.129:0/1569515087 >> [v2:172.21.15.157:6808/2444017632,v1:172.21.15.157:6809/2444017632] conn(0x7f3e88077a30 0x7f3e88079ed0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.084+0000 7f3ea6294700 1 -- 172.21.15.129:0/1569515087 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f3e8806de70 msgr2=0x7f3e88070330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T20:51:59.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.084+0000 7f3ea6294700 1 --2- 172.21.15.129:0/1569515087 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f3e8806de70 0x7f3e88070330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.084+0000 7f3ea6294700 1 -- 172.21.15.129:0/1569515087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ea0100ce0 msgr2=0x7f3ea0198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:59.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.084+0000 7f3ea6294700 1 --2- 172.21.15.129:0/1569515087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ea0100ce0 0x7f3ea0198950 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3e9400e9c0 tx=0x7f3e9400ecd0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.084+0000 7f3ea4831700 1 -- 172.21.15.129:0/1569515087 reap_dead start 2024-03-11T20:51:59.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.084+0000 7f3ea6294700 1 -- 172.21.15.129:0/1569515087 shutdown_connections 2024-03-11T20:51:59.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.084+0000 7f3ea6294700 1 -- 172.21.15.129:0/1569515087 >> 172.21.15.129:0/1569515087 conn(0x7f3ea0077c10 msgr2=0x7f3ea00ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:59.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.084+0000 7f3ea6294700 1 -- 172.21.15.129:0/1569515087 shutdown_connections 2024-03-11T20:51:59.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.084+0000 7f3ea6294700 1 -- 172.21.15.129:0/1569515087 wait complete. 2024-03-11T20:51:59.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.271+0000 7f0eb2b92700 1 -- 172.21.15.129:0/686904216 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eac1003d0 msgr2=0x7f0eac100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:59.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.271+0000 7f0eb2b92700 1 --2- 172.21.15.129:0/686904216 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eac1003d0 0x7f0eac100850 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0e980098f0 tx=0x7f0e98009c00 comp rx=0 tx=0).stop 2024-03-11T20:51:59.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.271+0000 7f0eb2b92700 1 -- 172.21.15.129:0/686904216 shutdown_connections 2024-03-11T20:51:59.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.271+0000 7f0eb2b92700 1 --2- 172.21.15.129:0/686904216 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0eac100d90 0x7f0eac10ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.271+0000 7f0eb2b92700 1 --2- 172.21.15.129:0/686904216 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eac106e50 0x7f0eac107230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.271+0000 7f0eb2b92700 1 --2- 172.21.15.129:0/686904216 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eac1003d0 0x7f0eac100850 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.271+0000 7f0eb2b92700 1 -- 172.21.15.129:0/686904216 >> 172.21.15.129:0/686904216 conn(0x7f0eac0772e0 msgr2=0x7f0eac0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:59.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.271+0000 7f0eb2b92700 1 -- 172.21.15.129:0/686904216 shutdown_connections 2024-03-11T20:51:59.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.271+0000 7f0eb2b92700 1 -- 172.21.15.129:0/686904216 wait complete. 2024-03-11T20:51:59.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.271+0000 7f0eb2b92700 1 Processor -- start 2024-03-11T20:51:59.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.271+0000 7f0eb2b92700 1 -- start start 2024-03-11T20:51:59.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb2b92700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eac1003d0 0x7f0eac1015b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:59.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb2b92700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0eac100d90 0x7f0eac101af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:59.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb2b92700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eac106e50 0x7f0eac102030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:59.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb2b92700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac111700 con 0x7f0eac1003d0 2024-03-11T20:51:59.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb2b92700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac111580 con 0x7f0eac100d90 2024-03-11T20:51:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb2b92700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac111880 con 0x7f0eac106e50 2024-03-11T20:51:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb112f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eac106e50 0x7f0eac102030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb092e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eac1003d0 0x7f0eac1015b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb092e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eac1003d0 0x7f0eac1015b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50762/0 (socket says 172.21.15.129:50762) 2024-03-11T20:51:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb112f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eac106e50 0x7f0eac102030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:34652/0 (socket says 172.21.15.129:34652) 2024-03-11T20:51:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb092e700 1 -- 172.21.15.129:0/4110659012 learned_addr learned my addr 172.21.15.129:0/4110659012 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:51:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eabfff700 1 --2- 172.21.15.129:0/4110659012 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0eac100d90 0x7f0eac101af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb112f700 1 -- 172.21.15.129:0/4110659012 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0eac100d90 msgr2=0x7f0eac101af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb112f700 1 --2- 172.21.15.129:0/4110659012 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0eac100d90 0x7f0eac101af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb112f700 1 -- 172.21.15.129:0/4110659012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eac1003d0 msgr2=0x7f0eac1015b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb112f700 1 --2- 172.21.15.129:0/4110659012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eac1003d0 0x7f0eac1015b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb112f700 1 -- 172.21.15.129:0/4110659012 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac1a6150 con 0x7f0eac106e50 2024-03-11T20:51:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb092e700 1 --2- 172.21.15.129:0/4110659012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eac1003d0 0x7f0eac1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:51:59.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb112f700 1 --2- 172.21.15.129:0/4110659012 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eac106e50 0x7f0eac102030 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4007ee0 tx=0x7f0ea400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:59.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0ea9ffb700 1 -- 172.21.15.129:0/4110659012 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea401a070 con 0x7f0eac106e50 2024-03-11T20:51:59.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0ea9ffb700 1 -- 172.21.15.129:0/4110659012 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0ea400dd20 con 0x7f0eac106e50 2024-03-11T20:51:59.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0eb2b92700 1 -- 172.21.15.129:0/4110659012 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eac1a6440 con 0x7f0eac106e50 2024-03-11T20:51:59.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.272+0000 7f0ea9ffb700 1 -- 172.21.15.129:0/4110659012 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4014600 con 0x7f0eac106e50 2024-03-11T20:51:59.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.273+0000 7f0eb2b92700 1 -- 172.21.15.129:0/4110659012 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eac1a69a0 con 0x7f0eac106e50 2024-03-11T20:51:59.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.273+0000 7f0eb2b92700 1 -- 172.21.15.129:0/4110659012 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0eac1052f0 con 0x7f0eac106e50 2024-03-11T20:51:59.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.274+0000 7f0ea9ffb700 1 -- 172.21.15.129:0/4110659012 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 17) v1 ==== 91820+0+0 (secure 0 0 0) 0x7f0ea401d070 con 0x7f0eac106e50 2024-03-11T20:51:59.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.274+0000 7f0ea9ffb700 1 --2- 172.21.15.129:0/4110659012 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f0e9c06de70 0x7f0e9c070330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:59.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.274+0000 7f0eb092e700 1 -- 172.21.15.129:0/4110659012 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f0e9c06de70 msgr2=0x7f0e9c070330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.129:6800/2866184010 2024-03-11T20:51:59.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.274+0000 7f0ea9ffb700 1 -- 172.21.15.129:0/4110659012 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7207+0+0 (secure 0 0 0) 0x7f0ea408dae0 con 0x7f0eac106e50 2024-03-11T20:51:59.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.275+0000 7f0eb092e700 1 --2- 172.21.15.129:0/4110659012 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f0e9c06de70 0x7f0e9c070330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T20:51:59.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.275+0000 7f0ea9ffb700 1 --2- 172.21.15.129:0/4110659012 >> [v2:172.21.15.129:6802/3618360833,v1:172.21.15.129:6803/3618360833] conn(0x7f0e9c077ab0 0x7f0e9c079f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:51:59.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.275+0000 7f0ea9ffb700 1 -- 172.21.15.129:0/4110659012 --> [v2:172.21.15.129:6802/3618360833,v1:172.21.15.129:6803/3618360833] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ea4017ec0 con 0x7f0e9c077ab0 2024-03-11T20:51:59.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.275+0000 7f0ea9ffb700 1 -- 172.21.15.129:0/4110659012 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ea40922c0 con 0x7f0eac106e50 2024-03-11T20:51:59.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.275+0000 7f0eabfff700 1 --2- 172.21.15.129:0/4110659012 >> [v2:172.21.15.129:6802/3618360833,v1:172.21.15.129:6803/3618360833] conn(0x7f0e9c077ab0 0x7f0e9c079f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:51:59.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.275+0000 7f0eabfff700 1 --2- 172.21.15.129:0/4110659012 >> [v2:172.21.15.129:6802/3618360833,v1:172.21.15.129:6803/3618360833] conn(0x7f0e9c077ab0 0x7f0e9c079f50 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:51:59.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.276+0000 7f0ea9ffb700 1 -- 172.21.15.129:0/4110659012 <== osd.0 v2:172.21.15.129:6802/3618360833 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0ea4017ec0 con 0x7f0e9c077ab0 2024-03-11T20:51:59.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.292+0000 7f0eb2b92700 1 -- 172.21.15.129:0/4110659012 --> [v2:172.21.15.129:6802/3618360833,v1:172.21.15.129:6803/3618360833] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0eac1036f0 con 0x7f0e9c077ab0 2024-03-11T20:51:59.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.293+0000 7f0ea9ffb700 1 -- 172.21.15.129:0/4110659012 <== osd.0 v2:172.21.15.129:6802/3618360833 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0eac1036f0 con 0x7f0e9c077ab0 2024-03-11T20:51:59.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.293+0000 7f0eb2b92700 1 -- 172.21.15.129:0/4110659012 >> [v2:172.21.15.129:6802/3618360833,v1:172.21.15.129:6803/3618360833] conn(0x7f0e9c077ab0 msgr2=0x7f0e9c079f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:59.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.293+0000 7f0eb2b92700 1 --2- 172.21.15.129:0/4110659012 >> [v2:172.21.15.129:6802/3618360833,v1:172.21.15.129:6803/3618360833] conn(0x7f0e9c077ab0 0x7f0e9c079f50 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.293+0000 7f0eb2b92700 1 -- 172.21.15.129:0/4110659012 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f0e9c06de70 msgr2=0x7f0e9c070330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T20:51:59.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.293+0000 7f0eb2b92700 1 --2- 172.21.15.129:0/4110659012 >> [v2:172.21.15.129:6800/2866184010,v1:172.21.15.129:6801/2866184010] conn(0x7f0e9c06de70 0x7f0e9c070330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.293+0000 7f0eb2b92700 1 -- 172.21.15.129:0/4110659012 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eac106e50 msgr2=0x7f0eac102030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:51:59.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.293+0000 7f0eb2b92700 1 --2- 172.21.15.129:0/4110659012 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f0eac106e50 0x7f0eac102030 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4007ee0 tx=0x7f0ea400d3b0 comp rx=0 tx=0).stop 2024-03-11T20:51:59.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.293+0000 7f0eb112f700 1 -- 172.21.15.129:0/4110659012 reap_dead start 2024-03-11T20:51:59.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.293+0000 7f0eb2b92700 1 -- 172.21.15.129:0/4110659012 shutdown_connections 2024-03-11T20:51:59.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.293+0000 7f0eb2b92700 1 -- 172.21.15.129:0/4110659012 >> 172.21.15.129:0/4110659012 conn(0x7f0eac0772e0 msgr2=0x7f0eac104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:51:59.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.293+0000 7f0eb2b92700 1 -- 172.21.15.129:0/4110659012 shutdown_connections 2024-03-11T20:51:59.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:51:59.293+0000 7f0eb2b92700 1 -- 172.21.15.129:0/4110659012 wait complete. 2024-03-11T20:51:59.978 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:51:59 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:51:59.764+0000 7f0e7985a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T20:51:59.985 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:51:59 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:51:59.557+0000 7fc24708f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T20:52:00.013 INFO:teuthology.orchestra.run.smithi129.stdout:163208757272 2024-03-11T20:52:00.014 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-11T20:52:00.447 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:52:00 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:52:00.132+0000 7fc24708f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T20:52:00.447 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:52:00 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:52:00.232+0000 7fc24708f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T20:52:00.447 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:52:00 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:52:00.451+0000 7fc24708f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T20:52:00.581 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-mon[30184]: Standby manager daemon x restarted 2024-03-11T20:52:00.581 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-mon[30184]: Standby manager daemon x started 2024-03-11T20:52:00.581 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-mon[30184]: from='mgr.? 172.21.15.157:0/3481510534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T20:52:00.581 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-mon[30184]: from='mgr.? 172.21.15.157:0/3481510534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T20:52:00.582 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-mon[30184]: from='mgr.? 172.21.15.157:0/3481510534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T20:52:00.582 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-mon[30184]: from='mgr.? 172.21.15.157:0/3481510534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T20:52:00.582 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:52:00.358+0000 7f0e7985a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T20:52:00.582 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:52:00.458+0000 7f0e7985a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T20:52:00.582 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-mon[42066]: Standby manager daemon x restarted 2024-03-11T20:52:00.582 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-mon[42066]: Standby manager daemon x started 2024-03-11T20:52:00.582 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-mon[42066]: from='mgr.? 172.21.15.157:0/3481510534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T20:52:00.582 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-mon[42066]: from='mgr.? 172.21.15.157:0/3481510534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T20:52:00.583 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-mon[42066]: from='mgr.? 172.21.15.157:0/3481510534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T20:52:00.583 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-mon[42066]: from='mgr.? 172.21.15.157:0/3481510534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T20:52:00.609 INFO:teuthology.orchestra.run.smithi129.stdout:128849018909 2024-03-11T20:52:00.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-11T20:52:00.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:00 smithi157 ceph-mon[33359]: Standby manager daemon x restarted 2024-03-11T20:52:00.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:00 smithi157 ceph-mon[33359]: Standby manager daemon x started 2024-03-11T20:52:00.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:00 smithi157 ceph-mon[33359]: from='mgr.? 172.21.15.157:0/3481510534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T20:52:00.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:00 smithi157 ceph-mon[33359]: from='mgr.? 172.21.15.157:0/3481510534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T20:52:00.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:00 smithi157 ceph-mon[33359]: from='mgr.? 172.21.15.157:0/3481510534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T20:52:00.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:00 smithi157 ceph-mon[33359]: from='mgr.? 172.21.15.157:0/3481510534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T20:52:00.736 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:52:00 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: [11/Mar/2024:20:52:00] ENGINE Bus STARTING 2024-03-11T20:52:00.736 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:52:00 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: CherryPy Checker: 2024-03-11T20:52:00.736 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:52:00 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: The Application mounted at '' has an empty config. 2024-03-11T20:52:00.736 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:52:00 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: 2024-03-11T20:52:00.737 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:52:00 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: [11/Mar/2024:20:52:00] ENGINE Serving on http://:::9283 2024-03-11T20:52:00.737 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 20:52:00 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x[35457]: [11/Mar/2024:20:52:00] ENGINE Bus STARTED 2024-03-11T20:52:00.839 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:52:00.687+0000 7f0e7985a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T20:52:00.839 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:52:00] ENGINE Bus STARTING 2024-03-11T20:52:01.104 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: CherryPy Checker: 2024-03-11T20:52:01.105 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: The Application mounted at '' has an empty config. 2024-03-11T20:52:01.105 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: 2024-03-11T20:52:01.105 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:52:00] ENGINE Serving on http://:::9283 2024-03-11T20:52:01.105 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:00 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:52:00] ENGINE Bus STARTED 2024-03-11T20:52:01.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.604+0000 7fae02ee1700 1 -- 172.21.15.129:0/2188536263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadfc1004c0 msgr2=0x7fadfc100940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:01.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.604+0000 7fae02ee1700 1 --2- 172.21.15.129:0/2188536263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadfc1004c0 0x7fadfc100940 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fade80099c0 tx=0x7fade8009cd0 comp rx=0 tx=0).stop 2024-03-11T20:52:01.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.604+0000 7fae02ee1700 1 -- 172.21.15.129:0/2188536263 shutdown_connections 2024-03-11T20:52:01.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.604+0000 7fae02ee1700 1 --2- 172.21.15.129:0/2188536263 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadfc100e80 0x7fadfc10d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:01.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.604+0000 7fae02ee1700 1 --2- 172.21.15.129:0/2188536263 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fadfc106570 0x7fadfc106950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:01.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.604+0000 7fae02ee1700 1 --2- 172.21.15.129:0/2188536263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadfc1004c0 0x7fadfc100940 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:01.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.604+0000 7fae02ee1700 1 -- 172.21.15.129:0/2188536263 >> 172.21.15.129:0/2188536263 conn(0x7fadfc0772e0 msgr2=0x7fadfc0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:01.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.604+0000 7fae02ee1700 1 -- 172.21.15.129:0/2188536263 shutdown_connections 2024-03-11T20:52:01.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.604+0000 7fae02ee1700 1 -- 172.21.15.129:0/2188536263 wait complete. 2024-03-11T20:52:01.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.605+0000 7fae02ee1700 1 Processor -- start 2024-03-11T20:52:01.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.605+0000 7fae02ee1700 1 -- start start 2024-03-11T20:52:01.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.605+0000 7fae02ee1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadfc1004c0 0x7fadfc19b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:01.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.605+0000 7fae02ee1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadfc100e80 0x7fadfc19bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:01.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.605+0000 7fae02ee1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fadfc106570 0x7fadfc1956d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:01.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.605+0000 7fae02ee1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc10f400 con 0x7fadfc1004c0 2024-03-11T20:52:01.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.605+0000 7fae02ee1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc10f280 con 0x7fadfc100e80 2024-03-11T20:52:01.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.605+0000 7fae02ee1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc10f580 con 0x7fadfc106570 2024-03-11T20:52:01.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.605+0000 7fae00c7d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadfc1004c0 0x7fadfc19b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:01.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.605+0000 7fae0147e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fadfc106570 0x7fadfc1956d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:01.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.605+0000 7fadfbfff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadfc100e80 0x7fadfc19bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:01.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.605+0000 7fae0147e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fadfc106570 0x7fadfc1956d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:34716/0 (socket says 172.21.15.129:34716) 2024-03-11T20:52:01.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.605+0000 7fadfbfff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadfc100e80 0x7fadfc19bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.129:40660/0 (socket says 172.21.15.129:40660) 2024-03-11T20:52:01.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.605+0000 7fae0147e700 1 -- 172.21.15.129:0/3973685965 learned_addr learned my addr 172.21.15.129:0/3973685965 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:52:01.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.606+0000 7fae0147e700 1 -- 172.21.15.129:0/3973685965 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadfc100e80 msgr2=0x7fadfc19bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:01.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.606+0000 7fae0147e700 1 --2- 172.21.15.129:0/3973685965 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadfc100e80 0x7fadfc19bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:01.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.606+0000 7fae0147e700 1 -- 172.21.15.129:0/3973685965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadfc1004c0 msgr2=0x7fadfc19b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:01.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.606+0000 7fae0147e700 1 --2- 172.21.15.129:0/3973685965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadfc1004c0 0x7fadfc19b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:01.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.606+0000 7fae0147e700 1 -- 172.21.15.129:0/3973685965 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc195fb0 con 0x7fadfc106570 2024-03-11T20:52:01.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.606+0000 7fae0147e700 1 --2- 172.21.15.129:0/3973685965 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fadfc106570 0x7fadfc1956d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fadf400a570 tx=0x7fadf400e3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:01.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.606+0000 7fadf9ffb700 1 -- 172.21.15.129:0/3973685965 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf4004db0 con 0x7fadfc106570 2024-03-11T20:52:01.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.606+0000 7fae02ee1700 1 -- 172.21.15.129:0/3973685965 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadfc1962a0 con 0x7fadfc106570 2024-03-11T20:52:01.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.606+0000 7fadf9ffb700 1 -- 172.21.15.129:0/3973685965 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fadf400ebe0 con 0x7fadfc106570 2024-03-11T20:52:01.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.606+0000 7fadf9ffb700 1 -- 172.21.15.129:0/3973685965 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf40044c0 con 0x7fadfc106570 2024-03-11T20:52:01.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.606+0000 7fae02ee1700 1 -- 172.21.15.129:0/3973685965 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadfc1a2580 con 0x7fadfc106570 2024-03-11T20:52:01.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.607+0000 7fae02ee1700 1 -- 172.21.15.129:0/3973685965 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fadfc105a20 con 0x7fadfc106570 2024-03-11T20:52:01.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.608+0000 7fadf9ffb700 1 -- 172.21.15.129:0/3973685965 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 19) v1 ==== 91436+0+0 (secure 0 0 0) 0x7fadf401d020 con 0x7fadfc106570 2024-03-11T20:52:01.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.608+0000 7fadf9ffb700 1 -- 172.21.15.129:0/3973685965 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7508+0+0 (secure 0 0 0) 0x7fadf408e480 con 0x7fadfc106570 2024-03-11T20:52:01.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.608+0000 7fadf9ffb700 1 --2- 172.21.15.129:0/3973685965 >> [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087] conn(0x7fadec074db0 0x7fadec077250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:01.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.609+0000 7fadf9ffb700 1 -- 172.21.15.129:0/3973685965 --> [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadec077920 con 0x7fadec074db0 2024-03-11T20:52:01.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.609+0000 7fadf9ffb700 1 -- 172.21.15.129:0/3973685965 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fadf4095050 con 0x7fadfc106570 2024-03-11T20:52:01.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.609+0000 7fae00c7d700 1 --2- 172.21.15.129:0/3973685965 >> [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087] conn(0x7fadec074db0 0x7fadec077250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:01.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.609+0000 7fae00c7d700 1 --2- 172.21.15.129:0/3973685965 >> [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087] conn(0x7fadec074db0 0x7fadec077250 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:01.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.610+0000 7fadf9ffb700 1 -- 172.21.15.129:0/3973685965 <== osd.2 v2:172.21.15.129:6818/1600233087 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fadec077920 con 0x7fadec074db0 2024-03-11T20:52:01.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.626+0000 7fae02ee1700 1 -- 172.21.15.129:0/3973685965 --> [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fadfc101be0 con 0x7fadec074db0 2024-03-11T20:52:01.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.626+0000 7fadf9ffb700 1 -- 172.21.15.129:0/3973685965 <== osd.2 v2:172.21.15.129:6818/1600233087 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fadfc101be0 con 0x7fadec074db0 2024-03-11T20:52:01.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.626+0000 7fae02ee1700 1 -- 172.21.15.129:0/3973685965 >> [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087] conn(0x7fadec074db0 msgr2=0x7fadec077250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:01.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.626+0000 7fae02ee1700 1 --2- 172.21.15.129:0/3973685965 >> [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087] conn(0x7fadec074db0 0x7fadec077250 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:01.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.626+0000 7fae02ee1700 1 -- 172.21.15.129:0/3973685965 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fadfc106570 msgr2=0x7fadfc1956d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:01.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.626+0000 7fae02ee1700 1 --2- 172.21.15.129:0/3973685965 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fadfc106570 0x7fadfc1956d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fadf400a570 tx=0x7fadf400e3b0 comp rx=0 tx=0).stop 2024-03-11T20:52:01.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.626+0000 7fae02ee1700 1 -- 172.21.15.129:0/3973685965 shutdown_connections 2024-03-11T20:52:01.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.626+0000 7fae02ee1700 1 --2- 172.21.15.129:0/3973685965 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fadfc106570 0x7fadfc1956d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:01.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.626+0000 7fae02ee1700 1 --2- 172.21.15.129:0/3973685965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadfc1004c0 0x7fadfc19b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:01.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.626+0000 7fae02ee1700 1 --2- 172.21.15.129:0/3973685965 >> [v2:172.21.15.129:6818/1600233087,v1:172.21.15.129:6819/1600233087] conn(0x7fadec074db0 0x7fadec077250 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:01.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.626+0000 7fae02ee1700 1 --2- 172.21.15.129:0/3973685965 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fadfc100e80 0x7fadfc19bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:01.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.626+0000 7fae02ee1700 1 -- 172.21.15.129:0/3973685965 >> 172.21.15.129:0/3973685965 conn(0x7fadfc0772e0 msgr2=0x7fadfc104070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:01.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.626+0000 7fae02ee1700 1 -- 172.21.15.129:0/3973685965 shutdown_connections 2024-03-11T20:52:01.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:01.626+0000 7fae02ee1700 1 -- 172.21.15.129:0/3973685965 wait complete. 2024-03-11T20:52:01.634 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-11T20:52:01.634 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: Active manager daemon y restarted 2024-03-11T20:52:01.634 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: Activating manager daemon y 2024-03-11T20:52:01.634 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T20:52:01.634 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: mgrmap e19: y(active, starting, since 0.00909141s), standbys: x 2024-03-11T20:52:01.634 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:52:01.634 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:52:01.634 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:52:01.635 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T20:52:01.635 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T20:52:01.635 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:52:01.635 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:52:01.635 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:52:01.635 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:52:01.635 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:52:01.635 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:52:01.635 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:52:01.635 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:52:01.635 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T20:52:01.635 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T20:52:01.635 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T20:52:01.636 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: Manager daemon y is now available 2024-03-11T20:52:01.636 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:52:01.636 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:52:01.636 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:01.636 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:01.636 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T20:52:01.636 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T20:52:01.636 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-11T20:52:01.636 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: Active manager daemon y restarted 2024-03-11T20:52:01.636 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: Activating manager daemon y 2024-03-11T20:52:01.636 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T20:52:01.637 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: mgrmap e19: y(active, starting, since 0.00909141s), standbys: x 2024-03-11T20:52:01.637 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:52:01.637 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:52:01.637 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:52:01.637 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T20:52:01.637 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T20:52:01.637 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:52:01.637 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:52:01.637 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:52:01.637 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:52:01.637 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:52:01.637 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:52:01.638 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:52:01.638 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:52:01.638 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T20:52:01.638 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T20:52:01.638 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T20:52:01.638 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: Manager daemon y is now available 2024-03-11T20:52:01.638 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:52:01.638 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:52:01.638 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:01.638 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:01.638 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T20:52:01.638 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T20:52:01.928 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-11T20:52:01.928 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: Active manager daemon y restarted 2024-03-11T20:52:01.928 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: Activating manager daemon y 2024-03-11T20:52:01.928 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T20:52:01.928 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: mgrmap e19: y(active, starting, since 0.00909141s), standbys: x 2024-03-11T20:52:01.928 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T20:52:01.928 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T20:52:01.928 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T20:52:01.928 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T20:52:01.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T20:52:01.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T20:52:01.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T20:52:01.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T20:52:01.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T20:52:01.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T20:52:01.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T20:52:01.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T20:52:01.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T20:52:01.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T20:52:01.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T20:52:01.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T20:52:01.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: Manager daemon y is now available 2024-03-11T20:52:01.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:52:01.929 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:52:01.930 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:01.930 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:01.930 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T20:52:01.930 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T20:52:02.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.547+0000 7f9adbeb2700 1 -- 172.21.15.129:0/4053847502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad4106040 msgr2=0x7f9ad410dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:02.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.547+0000 7f9adbeb2700 1 --2- 172.21.15.129:0/4053847502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad4106040 0x7f9ad410dc10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9acc009fc0 tx=0x7f9acc00b330 comp rx=0 tx=0).stop 2024-03-11T20:52:02.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.547+0000 7f9adbeb2700 1 -- 172.21.15.129:0/4053847502 shutdown_connections 2024-03-11T20:52:02.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.547+0000 7f9adbeb2700 1 --2- 172.21.15.129:0/4053847502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad4106040 0x7f9ad410dc10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:02.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.547+0000 7f9adbeb2700 1 --2- 172.21.15.129:0/4053847502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9ad4100f60 0x7f9ad4101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:02.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.547+0000 7f9adbeb2700 1 --2- 172.21.15.129:0/4053847502 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ad4101910 0x7f9ad4105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:02.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.547+0000 7f9adbeb2700 1 -- 172.21.15.129:0/4053847502 >> 172.21.15.129:0/4053847502 conn(0x7f9ad4077de0 msgr2=0x7f9ad40781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:02.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.548+0000 7f9adbeb2700 1 -- 172.21.15.129:0/4053847502 shutdown_connections 2024-03-11T20:52:02.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.548+0000 7f9adbeb2700 1 -- 172.21.15.129:0/4053847502 wait complete. 2024-03-11T20:52:02.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.548+0000 7f9adbeb2700 1 Processor -- start 2024-03-11T20:52:02.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.548+0000 7f9adbeb2700 1 -- start start 2024-03-11T20:52:02.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.548+0000 7f9adbeb2700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad4100f60 0x7f9ad4198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:02.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.548+0000 7f9adbeb2700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ad4101910 0x7f9ad4198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:02.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.548+0000 7f9adbeb2700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9ad4106040 0x7f9ad41a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:02.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.548+0000 7f9adbeb2700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad4110550 con 0x7f9ad4100f60 2024-03-11T20:52:02.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.548+0000 7f9adbeb2700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad41103d0 con 0x7f9ad4106040 2024-03-11T20:52:02.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.548+0000 7f9adbeb2700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad41106d0 con 0x7f9ad4101910 2024-03-11T20:52:02.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.548+0000 7f9ad944d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ad4101910 0x7f9ad4198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:02.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.548+0000 7f9ad9c4e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad4100f60 0x7f9ad4198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:02.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.548+0000 7f9ad944d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ad4101910 0x7f9ad4198e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:34744/0 (socket says 172.21.15.129:34744) 2024-03-11T20:52:02.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.548+0000 7f9ad944d700 1 -- 172.21.15.129:0/1264875063 learned_addr learned my addr 172.21.15.129:0/1264875063 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:52:02.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.548+0000 7f9ada44f700 1 --2- 172.21.15.129:0/1264875063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9ad4106040 0x7f9ad41a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:02.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.549+0000 7f9ad944d700 1 -- 172.21.15.129:0/1264875063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9ad4106040 msgr2=0x7f9ad41a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:02.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.549+0000 7f9ad944d700 1 --2- 172.21.15.129:0/1264875063 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9ad4106040 0x7f9ad41a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:02.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.549+0000 7f9ad944d700 1 -- 172.21.15.129:0/1264875063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad4100f60 msgr2=0x7f9ad4198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:02.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.549+0000 7f9ad944d700 1 --2- 172.21.15.129:0/1264875063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad4100f60 0x7f9ad4198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:02.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.549+0000 7f9ad944d700 1 -- 172.21.15.129:0/1264875063 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad41a3ef0 con 0x7f9ad4101910 2024-03-11T20:52:02.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.549+0000 7f9ad9c4e700 1 --2- 172.21.15.129:0/1264875063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad4100f60 0x7f9ad4198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:52:02.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.549+0000 7f9ad944d700 1 --2- 172.21.15.129:0/1264875063 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ad4101910 0x7f9ad4198e70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400ec70 tx=0x7f9ac400ef80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:02.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.549+0000 7f9acaffd700 1 -- 172.21.15.129:0/1264875063 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac400cc90 con 0x7f9ad4101910 2024-03-11T20:52:02.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.549+0000 7f9acaffd700 1 -- 172.21.15.129:0/1264875063 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9ac400ce30 con 0x7f9ad4101910 2024-03-11T20:52:02.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.549+0000 7f9adbeb2700 1 -- 172.21.15.129:0/1264875063 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad41a40c0 con 0x7f9ad4101910 2024-03-11T20:52:02.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.549+0000 7f9acaffd700 1 -- 172.21.15.129:0/1264875063 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4003df0 con 0x7f9ad4101910 2024-03-11T20:52:02.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.549+0000 7f9adbeb2700 1 -- 172.21.15.129:0/1264875063 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad41a4600 con 0x7f9ad4101910 2024-03-11T20:52:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.550+0000 7f9adbeb2700 1 -- 172.21.15.129:0/1264875063 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9ad40624f0 con 0x7f9ad4101910 2024-03-11T20:52:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.550+0000 7f9acaffd700 1 -- 172.21.15.129:0/1264875063 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 20) v1 ==== 91563+0+0 (secure 0 0 0) 0x7f9ac4004000 con 0x7f9ad4101910 2024-03-11T20:52:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.551+0000 7f9acaffd700 1 --2- 172.21.15.129:0/1264875063 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f9ac006db90 0x7f9ac0070050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.551+0000 7f9ad9c4e700 1 --2- 172.21.15.129:0/1264875063 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f9ac006db90 0x7f9ac0070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.551+0000 7f9acaffd700 1 -- 172.21.15.129:0/1264875063 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f9ac408dfe0 con 0x7f9ad4101910 2024-03-11T20:52:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.551+0000 7f9acaffd700 1 --2- 172.21.15.129:0/1264875063 >> [v2:172.21.15.157:6824/1786686696,v1:172.21.15.157:6825/1786686696] conn(0x7f9ac0077a20 0x7f9ac0079ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.551+0000 7f9acaffd700 1 -- 172.21.15.129:0/1264875063 --> [v2:172.21.15.157:6824/1786686696,v1:172.21.15.157:6825/1786686696] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ac007a590 con 0x7f9ac0077a20 2024-03-11T20:52:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.551+0000 7f9acaffd700 1 -- 172.21.15.129:0/1264875063 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ac4095050 con 0x7f9ad4101910 2024-03-11T20:52:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.551+0000 7f9ad9c4e700 1 --2- 172.21.15.129:0/1264875063 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f9ac006db90 0x7f9ac0070050 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0007fd0 tx=0x7f9ad00073d0 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.551+0000 7f9ada44f700 1 --2- 172.21.15.129:0/1264875063 >> [v2:172.21.15.157:6824/1786686696,v1:172.21.15.157:6825/1786686696] conn(0x7f9ac0077a20 0x7f9ac0079ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.552+0000 7f9ada44f700 1 --2- 172.21.15.129:0/1264875063 >> [v2:172.21.15.157:6824/1786686696,v1:172.21.15.157:6825/1786686696] conn(0x7f9ac0077a20 0x7f9ac0079ec0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.553+0000 7f9acaffd700 1 -- 172.21.15.129:0/1264875063 <== osd.7 v2:172.21.15.157:6824/1786686696 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9ac007a590 con 0x7f9ac0077a20 2024-03-11T20:52:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.569+0000 7f9adbeb2700 1 -- 172.21.15.129:0/1264875063 --> [v2:172.21.15.157:6824/1786686696,v1:172.21.15.157:6825/1786686696] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9ad40628f0 con 0x7f9ac0077a20 2024-03-11T20:52:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.569+0000 7f9acaffd700 1 -- 172.21.15.129:0/1264875063 <== osd.7 v2:172.21.15.157:6824/1786686696 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9ad40628f0 con 0x7f9ac0077a20 2024-03-11T20:52:02.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.570+0000 7f9adbeb2700 1 -- 172.21.15.129:0/1264875063 >> [v2:172.21.15.157:6824/1786686696,v1:172.21.15.157:6825/1786686696] conn(0x7f9ac0077a20 msgr2=0x7f9ac0079ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:02.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.570+0000 7f9adbeb2700 1 --2- 172.21.15.129:0/1264875063 >> [v2:172.21.15.157:6824/1786686696,v1:172.21.15.157:6825/1786686696] conn(0x7f9ac0077a20 0x7f9ac0079ec0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:02.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.570+0000 7f9adbeb2700 1 -- 172.21.15.129:0/1264875063 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f9ac006db90 msgr2=0x7f9ac0070050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:02.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.570+0000 7f9adbeb2700 1 --2- 172.21.15.129:0/1264875063 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f9ac006db90 0x7f9ac0070050 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0007fd0 tx=0x7f9ad00073d0 comp rx=0 tx=0).stop 2024-03-11T20:52:02.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.570+0000 7f9adbeb2700 1 -- 172.21.15.129:0/1264875063 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ad4101910 msgr2=0x7f9ad4198e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:02.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.570+0000 7f9adbeb2700 1 --2- 172.21.15.129:0/1264875063 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9ad4101910 0x7f9ad4198e70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400ec70 tx=0x7f9ac400ef80 comp rx=0 tx=0).stop 2024-03-11T20:52:02.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.570+0000 7f9ada44f700 1 -- 172.21.15.129:0/1264875063 reap_dead start 2024-03-11T20:52:02.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.570+0000 7f9adbeb2700 1 -- 172.21.15.129:0/1264875063 shutdown_connections 2024-03-11T20:52:02.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.570+0000 7f9adbeb2700 1 -- 172.21.15.129:0/1264875063 >> 172.21.15.129:0/1264875063 conn(0x7f9ad4077de0 msgr2=0x7f9ad40fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:02.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.570+0000 7f9adbeb2700 1 -- 172.21.15.129:0/1264875063 shutdown_connections 2024-03-11T20:52:02.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:02.570+0000 7f9adbeb2700 1 -- 172.21.15.129:0/1264875063 wait complete. 2024-03-11T20:52:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:02 smithi129 ceph-mon[30184]: mgrmap e20: y(active, since 1.01169s), standbys: x 2024-03-11T20:52:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:02 smithi129 ceph-mon[30184]: [11/Mar/2024:20:52:02] ENGINE Bus STARTING 2024-03-11T20:52:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:02 smithi129 ceph-mon[30184]: [11/Mar/2024:20:52:02] ENGINE Serving on http://172.21.15.129:8765 2024-03-11T20:52:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:02 smithi129 ceph-mon[30184]: [11/Mar/2024:20:52:02] ENGINE Serving on https://172.21.15.129:7150 2024-03-11T20:52:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:02 smithi129 ceph-mon[30184]: [11/Mar/2024:20:52:02] ENGINE Bus STARTED 2024-03-11T20:52:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:02 smithi129 ceph-mon[42066]: mgrmap e20: y(active, since 1.01169s), standbys: x 2024-03-11T20:52:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:02 smithi129 ceph-mon[42066]: [11/Mar/2024:20:52:02] ENGINE Bus STARTING 2024-03-11T20:52:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:02 smithi129 ceph-mon[42066]: [11/Mar/2024:20:52:02] ENGINE Serving on http://172.21.15.129:8765 2024-03-11T20:52:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:02 smithi129 ceph-mon[42066]: [11/Mar/2024:20:52:02] ENGINE Serving on https://172.21.15.129:7150 2024-03-11T20:52:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:02 smithi129 ceph-mon[42066]: [11/Mar/2024:20:52:02] ENGINE Bus STARTED 2024-03-11T20:52:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:02 smithi157 ceph-mon[33359]: mgrmap e20: y(active, since 1.01169s), standbys: x 2024-03-11T20:52:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:02 smithi157 ceph-mon[33359]: [11/Mar/2024:20:52:02] ENGINE Bus STARTING 2024-03-11T20:52:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:02 smithi157 ceph-mon[33359]: [11/Mar/2024:20:52:02] ENGINE Serving on http://172.21.15.129:8765 2024-03-11T20:52:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:02 smithi157 ceph-mon[33359]: [11/Mar/2024:20:52:02] ENGINE Serving on https://172.21.15.129:7150 2024-03-11T20:52:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:02 smithi157 ceph-mon[33359]: [11/Mar/2024:20:52:02] ENGINE Bus STARTED 2024-03-11T20:52:03.345 INFO:teuthology.orchestra.run.smithi129.stdout:231928233999 2024-03-11T20:52:03.346 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-11T20:52:03.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:52:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:52:03.584 INFO:teuthology.orchestra.run.smithi129.stdout:197568495636 2024-03-11T20:52:03.584 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-11T20:52:03.907 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:03 smithi129 ceph-mon[30184]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:52:03.907 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:03 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:03.907 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:03 smithi129 ceph-mon[42066]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:52:03.908 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:03 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:03.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:03 smithi157 ceph-mon[33359]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T20:52:03.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:03 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:04.359 INFO:teuthology.orchestra.run.smithi129.stdout:77309411367 2024-03-11T20:52:04.359 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T20:52:04.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.563+0000 7f2e82760700 1 -- 172.21.15.129:0/2233721576 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e7c100fb0 msgr2=0x7f2e7c101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:04.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.563+0000 7f2e82760700 1 --2- 172.21.15.129:0/2233721576 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e7c100fb0 0x7f2e7c101390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c0099c0 tx=0x7f2e6c009cd0 comp rx=0 tx=0).stop 2024-03-11T20:52:04.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.563+0000 7f2e82760700 1 -- 172.21.15.129:0/2233721576 shutdown_connections 2024-03-11T20:52:04.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.563+0000 7f2e82760700 1 --2- 172.21.15.129:0/2233721576 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e7c106090 0x7f2e7c10dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:04.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.563+0000 7f2e82760700 1 --2- 172.21.15.129:0/2233721576 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e7c100fb0 0x7f2e7c101390 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:04.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.563+0000 7f2e82760700 1 --2- 172.21.15.129:0/2233721576 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e7c101960 0x7f2e7c1059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:04.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.563+0000 7f2e82760700 1 -- 172.21.15.129:0/2233721576 >> 172.21.15.129:0/2233721576 conn(0x7f2e7c077e10 msgr2=0x7f2e7c078220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:04.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.564+0000 7f2e82760700 1 -- 172.21.15.129:0/2233721576 shutdown_connections 2024-03-11T20:52:04.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.564+0000 7f2e82760700 1 -- 172.21.15.129:0/2233721576 wait complete. 2024-03-11T20:52:04.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.564+0000 7f2e82760700 1 Processor -- start 2024-03-11T20:52:04.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.564+0000 7f2e82760700 1 -- start start 2024-03-11T20:52:04.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.564+0000 7f2e82760700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e7c100fb0 0x7f2e7c102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:04.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e82760700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e7c101960 0x7f2e7c102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:04.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e82760700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e7c106090 0x7f2e7c1034d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:04.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e82760700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c1105a0 con 0x7f2e7c106090 2024-03-11T20:52:04.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e82760700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c110420 con 0x7f2e7c100fb0 2024-03-11T20:52:04.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e82760700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7c110720 con 0x7f2e7c101960 2024-03-11T20:52:04.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e80cfd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e7c106090 0x7f2e7c1034d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:04.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e7b7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e7c101960 0x7f2e7c102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:04.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e80cfd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e7c106090 0x7f2e7c1034d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50876/0 (socket says 172.21.15.129:50876) 2024-03-11T20:52:04.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e80cfd700 1 -- 172.21.15.129:0/1275273439 learned_addr learned my addr 172.21.15.129:0/1275273439 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:52:04.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e7bfff700 1 --2- 172.21.15.129:0/1275273439 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e7c100fb0 0x7f2e7c102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:04.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e80cfd700 1 -- 172.21.15.129:0/1275273439 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e7c101960 msgr2=0x7f2e7c102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:04.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e80cfd700 1 --2- 172.21.15.129:0/1275273439 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e7c101960 0x7f2e7c102f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:04.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e80cfd700 1 -- 172.21.15.129:0/1275273439 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e7c100fb0 msgr2=0x7f2e7c102a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:04.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e80cfd700 1 --2- 172.21.15.129:0/1275273439 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e7c100fb0 0x7f2e7c102a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:04.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e80cfd700 1 -- 172.21.15.129:0/1275273439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c1a6110 con 0x7f2e7c106090 2024-03-11T20:52:04.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e7b7fe700 1 --2- 172.21.15.129:0/1275273439 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f2e7c101960 0x7f2e7c102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:52:04.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.565+0000 7f2e80cfd700 1 --2- 172.21.15.129:0/1275273439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e7c106090 0x7f2e7c1034d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400c370 tx=0x7f2e7400c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:04.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.566+0000 7f2e797fa700 1 -- 172.21.15.129:0/1275273439 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e740092e0 con 0x7f2e7c106090 2024-03-11T20:52:04.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.566+0000 7f2e797fa700 1 -- 172.21.15.129:0/1275273439 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2e74003a00 con 0x7f2e7c106090 2024-03-11T20:52:04.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.566+0000 7f2e82760700 1 -- 172.21.15.129:0/1275273439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7c1a63a0 con 0x7f2e7c106090 2024-03-11T20:52:04.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.566+0000 7f2e797fa700 1 -- 172.21.15.129:0/1275273439 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e740045e0 con 0x7f2e7c106090 2024-03-11T20:52:04.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.566+0000 7f2e82760700 1 -- 172.21.15.129:0/1275273439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e7c1a6890 con 0x7f2e7c106090 2024-03-11T20:52:04.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.567+0000 7f2e82760700 1 -- 172.21.15.129:0/1275273439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2e7c0624f0 con 0x7f2e7c106090 2024-03-11T20:52:04.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.567+0000 7f2e797fa700 1 -- 172.21.15.129:0/1275273439 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 21) v1 ==== 91757+0+0 (secure 0 0 0) 0x7f2e7401a020 con 0x7f2e7c106090 2024-03-11T20:52:04.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.568+0000 7f2e797fa700 1 --2- 172.21.15.129:0/1275273439 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f2e6406ddf0 0x7f2e640702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:04.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.568+0000 7f2e797fa700 1 -- 172.21.15.129:0/1275273439 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f2e7408d9d0 con 0x7f2e7c106090 2024-03-11T20:52:04.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.568+0000 7f2e797fa700 1 --2- 172.21.15.129:0/1275273439 >> [v2:172.21.15.129:6810/2519234527,v1:172.21.15.129:6811/2519234527] conn(0x7f2e64077c80 0x7f2e6407a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:04.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.568+0000 7f2e797fa700 1 -- 172.21.15.129:0/1275273439 --> [v2:172.21.15.129:6810/2519234527,v1:172.21.15.129:6811/2519234527] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e7401fe50 con 0x7f2e64077c80 2024-03-11T20:52:04.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.568+0000 7f2e797fa700 1 -- 172.21.15.129:0/1275273439 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e740922a0 con 0x7f2e7c106090 2024-03-11T20:52:04.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.568+0000 7f2e7b7fe700 1 --2- 172.21.15.129:0/1275273439 >> [v2:172.21.15.129:6810/2519234527,v1:172.21.15.129:6811/2519234527] conn(0x7f2e64077c80 0x7f2e6407a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:04.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.568+0000 7f2e7bfff700 1 --2- 172.21.15.129:0/1275273439 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f2e6406ddf0 0x7f2e640702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:04.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.569+0000 7f2e7b7fe700 1 --2- 172.21.15.129:0/1275273439 >> [v2:172.21.15.129:6810/2519234527,v1:172.21.15.129:6811/2519234527] conn(0x7f2e64077c80 0x7f2e6407a120 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:04.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.569+0000 7f2e7bfff700 1 --2- 172.21.15.129:0/1275273439 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f2e6406ddf0 0x7f2e640702b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c003760 tx=0x7f2e6c005fb0 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:04.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.569+0000 7f2e797fa700 1 -- 172.21.15.129:0/1275273439 <== osd.1 v2:172.21.15.129:6810/2519234527 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2e7401fe50 con 0x7f2e64077c80 2024-03-11T20:52:04.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.588+0000 7f2e82760700 1 -- 172.21.15.129:0/1275273439 --> [v2:172.21.15.129:6810/2519234527,v1:172.21.15.129:6811/2519234527] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2e7c0628f0 con 0x7f2e64077c80 2024-03-11T20:52:04.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.588+0000 7f2e797fa700 1 -- 172.21.15.129:0/1275273439 <== osd.1 v2:172.21.15.129:6810/2519234527 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2e7c0628f0 con 0x7f2e64077c80 2024-03-11T20:52:04.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.589+0000 7f2e82760700 1 -- 172.21.15.129:0/1275273439 >> [v2:172.21.15.129:6810/2519234527,v1:172.21.15.129:6811/2519234527] conn(0x7f2e64077c80 msgr2=0x7f2e6407a120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:04.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.589+0000 7f2e82760700 1 --2- 172.21.15.129:0/1275273439 >> [v2:172.21.15.129:6810/2519234527,v1:172.21.15.129:6811/2519234527] conn(0x7f2e64077c80 0x7f2e6407a120 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:04.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.589+0000 7f2e82760700 1 -- 172.21.15.129:0/1275273439 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f2e6406ddf0 msgr2=0x7f2e640702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:04.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.589+0000 7f2e82760700 1 --2- 172.21.15.129:0/1275273439 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f2e6406ddf0 0x7f2e640702b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c003760 tx=0x7f2e6c005fb0 comp rx=0 tx=0).stop 2024-03-11T20:52:04.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.589+0000 7f2e82760700 1 -- 172.21.15.129:0/1275273439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e7c106090 msgr2=0x7f2e7c1034d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:04.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.589+0000 7f2e82760700 1 --2- 172.21.15.129:0/1275273439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e7c106090 0x7f2e7c1034d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400c370 tx=0x7f2e7400c680 comp rx=0 tx=0).stop 2024-03-11T20:52:04.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.589+0000 7f2e80cfd700 1 -- 172.21.15.129:0/1275273439 reap_dead start 2024-03-11T20:52:04.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.589+0000 7f2e82760700 1 -- 172.21.15.129:0/1275273439 shutdown_connections 2024-03-11T20:52:04.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.589+0000 7f2e82760700 1 -- 172.21.15.129:0/1275273439 >> 172.21.15.129:0/1275273439 conn(0x7f2e7c077e10 msgr2=0x7f2e7c104fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:04.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.589+0000 7f2e82760700 1 -- 172.21.15.129:0/1275273439 shutdown_connections 2024-03-11T20:52:04.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.589+0000 7f2e82760700 1 -- 172.21.15.129:0/1275273439 wait complete. 2024-03-11T20:52:04.891 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:04 smithi129 ceph-mon[30184]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-11T20:52:04.891 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:04.891 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:04.891 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:04 smithi129 ceph-mon[42066]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-11T20:52:04.891 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:04.891 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:04.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.894+0000 7fb88c4e1700 1 -- 172.21.15.129:0/4246233306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb88406a480 msgr2=0x7fb88406a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:04.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.894+0000 7fb88c4e1700 1 --2- 172.21.15.129:0/4246233306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb88406a480 0x7fb88406a900 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb8740098f0 tx=0x7fb874009c00 comp rx=0 tx=0).stop 2024-03-11T20:52:04.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.895+0000 7fb88c4e1700 1 -- 172.21.15.129:0/4246233306 shutdown_connections 2024-03-11T20:52:04.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.895+0000 7fb88c4e1700 1 --2- 172.21.15.129:0/4246233306 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb88406ae40 0x7fb88410ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:04.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.895+0000 7fb88c4e1700 1 --2- 172.21.15.129:0/4246233306 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb884107ed0 0x7fb8841082b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:04.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.895+0000 7fb88c4e1700 1 --2- 172.21.15.129:0/4246233306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb88406a480 0x7fb88406a900 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:04.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.895+0000 7fb88c4e1700 1 -- 172.21.15.129:0/4246233306 >> 172.21.15.129:0/4246233306 conn(0x7fb884077ba0 msgr2=0x7fb884077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:04.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.895+0000 7fb88c4e1700 1 -- 172.21.15.129:0/4246233306 shutdown_connections 2024-03-11T20:52:04.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.895+0000 7fb88c4e1700 1 -- 172.21.15.129:0/4246233306 wait complete. 2024-03-11T20:52:04.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.895+0000 7fb88c4e1700 1 Processor -- start 2024-03-11T20:52:04.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.895+0000 7fb88c4e1700 1 -- start start 2024-03-11T20:52:04.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88c4e1700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb88406a480 0x7fb884101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:04.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88c4e1700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb88406ae40 0x7fb8841015d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:04.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88c4e1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb884107ed0 0x7fb884101b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:04.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88c4e1700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8841115e0 con 0x7fb88406a480 2024-03-11T20:52:04.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88c4e1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb884111460 con 0x7fb884107ed0 2024-03-11T20:52:04.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88c4e1700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb884111760 con 0x7fb88406ae40 2024-03-11T20:52:04.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88a27d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb88406a480 0x7fb884101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:04.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88a27d700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb88406a480 0x7fb884101090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50888/0 (socket says 172.21.15.129:50888) 2024-03-11T20:52:04.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88a27d700 1 -- 172.21.15.129:0/538091851 learned_addr learned my addr 172.21.15.129:0/538091851 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:52:04.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88aa7e700 1 --2- 172.21.15.129:0/538091851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb884107ed0 0x7fb884101b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:04.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb889a7c700 1 --2- 172.21.15.129:0/538091851 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb88406ae40 0x7fb8841015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:04.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88aa7e700 1 -- 172.21.15.129:0/538091851 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb88406ae40 msgr2=0x7fb8841015d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:04.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88aa7e700 1 --2- 172.21.15.129:0/538091851 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb88406ae40 0x7fb8841015d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:04.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88aa7e700 1 -- 172.21.15.129:0/538091851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb88406a480 msgr2=0x7fb884101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:04.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88aa7e700 1 --2- 172.21.15.129:0/538091851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb88406a480 0x7fb884101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:04.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88aa7e700 1 -- 172.21.15.129:0/538091851 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8841a6170 con 0x7fb884107ed0 2024-03-11T20:52:04.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88a27d700 1 --2- 172.21.15.129:0/538091851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb88406a480 0x7fb884101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:52:04.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.896+0000 7fb88aa7e700 1 --2- 172.21.15.129:0/538091851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb884107ed0 0x7fb884101b10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb87c00efc0 tx=0x7fb87c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:04.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.897+0000 7fb87b7fe700 1 -- 172.21.15.129:0/538091851 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c0092e0 con 0x7fb884107ed0 2024-03-11T20:52:04.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.897+0000 7fb87b7fe700 1 -- 172.21.15.129:0/538091851 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb87c004bb0 con 0x7fb884107ed0 2024-03-11T20:52:04.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.897+0000 7fb88c4e1700 1 -- 172.21.15.129:0/538091851 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8841a6400 con 0x7fb884107ed0 2024-03-11T20:52:04.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.897+0000 7fb87b7fe700 1 -- 172.21.15.129:0/538091851 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c00ce40 con 0x7fb884107ed0 2024-03-11T20:52:04.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.897+0000 7fb88c4e1700 1 -- 172.21.15.129:0/538091851 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8841a68c0 con 0x7fb884107ed0 2024-03-11T20:52:04.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.898+0000 7fb88c4e1700 1 -- 172.21.15.129:0/538091851 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb8840fff80 con 0x7fb884107ed0 2024-03-11T20:52:04.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.898+0000 7fb87b7fe700 1 -- 172.21.15.129:0/538091851 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 21) v1 ==== 91757+0+0 (secure 0 0 0) 0x7fb87c01b030 con 0x7fb884107ed0 2024-03-11T20:52:04.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.898+0000 7fb87b7fe700 1 --2- 172.21.15.129:0/538091851 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fb87006ddf0 0x7fb8700702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:04.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.899+0000 7fb87b7fe700 1 -- 172.21.15.129:0/538091851 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7508+0+0 (secure 0 0 0) 0x7fb87c08dc60 con 0x7fb884107ed0 2024-03-11T20:52:04.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.899+0000 7fb88a27d700 1 --2- 172.21.15.129:0/538091851 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fb87006ddf0 0x7fb8700702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:04.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.899+0000 7fb87b7fe700 1 --2- 172.21.15.129:0/538091851 >> [v2:172.21.15.129:6826/2265856698,v1:172.21.15.129:6827/2265856698] conn(0x7fb870077c80 0x7fb87007a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:04.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.899+0000 7fb87b7fe700 1 -- 172.21.15.129:0/538091851 --> [v2:172.21.15.129:6826/2265856698,v1:172.21.15.129:6827/2265856698] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb87007a7f0 con 0x7fb870077c80 2024-03-11T20:52:04.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.899+0000 7fb87b7fe700 1 -- 172.21.15.129:0/538091851 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb87c0922a0 con 0x7fb884107ed0 2024-03-11T20:52:04.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.899+0000 7fb889a7c700 1 --2- 172.21.15.129:0/538091851 >> [v2:172.21.15.129:6826/2265856698,v1:172.21.15.129:6827/2265856698] conn(0x7fb870077c80 0x7fb87007a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:04.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.899+0000 7fb88a27d700 1 --2- 172.21.15.129:0/538091851 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fb87006ddf0 0x7fb8700702b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb880009750 tx=0x7fb880009310 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:04.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.899+0000 7fb889a7c700 1 --2- 172.21.15.129:0/538091851 >> [v2:172.21.15.129:6826/2265856698,v1:172.21.15.129:6827/2265856698] conn(0x7fb870077c80 0x7fb87007a120 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:04.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.901+0000 7fb87b7fe700 1 -- 172.21.15.129:0/538091851 <== osd.3 v2:172.21.15.129:6826/2265856698 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb87007a7f0 con 0x7fb870077c80 2024-03-11T20:52:04.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.917+0000 7fb88c4e1700 1 -- 172.21.15.129:0/538091851 --> [v2:172.21.15.129:6826/2265856698,v1:172.21.15.129:6827/2265856698] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb884103520 con 0x7fb870077c80 2024-03-11T20:52:04.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.918+0000 7fb87b7fe700 1 -- 172.21.15.129:0/538091851 <== osd.3 v2:172.21.15.129:6826/2265856698 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb884103520 con 0x7fb870077c80 2024-03-11T20:52:04.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.918+0000 7fb88c4e1700 1 -- 172.21.15.129:0/538091851 >> [v2:172.21.15.129:6826/2265856698,v1:172.21.15.129:6827/2265856698] conn(0x7fb870077c80 msgr2=0x7fb87007a120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:04.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.918+0000 7fb88c4e1700 1 --2- 172.21.15.129:0/538091851 >> [v2:172.21.15.129:6826/2265856698,v1:172.21.15.129:6827/2265856698] conn(0x7fb870077c80 0x7fb87007a120 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:04.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.918+0000 7fb88c4e1700 1 -- 172.21.15.129:0/538091851 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fb87006ddf0 msgr2=0x7fb8700702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:04.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.918+0000 7fb88c4e1700 1 --2- 172.21.15.129:0/538091851 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fb87006ddf0 0x7fb8700702b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb880009750 tx=0x7fb880009310 comp rx=0 tx=0).stop 2024-03-11T20:52:04.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.918+0000 7fb88c4e1700 1 -- 172.21.15.129:0/538091851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb884107ed0 msgr2=0x7fb884101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:04.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.918+0000 7fb88c4e1700 1 --2- 172.21.15.129:0/538091851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb884107ed0 0x7fb884101b10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb87c00efc0 tx=0x7fb87c00c3a0 comp rx=0 tx=0).stop 2024-03-11T20:52:04.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.918+0000 7fb88aa7e700 1 -- 172.21.15.129:0/538091851 reap_dead start 2024-03-11T20:52:04.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.918+0000 7fb88c4e1700 1 -- 172.21.15.129:0/538091851 shutdown_connections 2024-03-11T20:52:04.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.918+0000 7fb88c4e1700 1 -- 172.21.15.129:0/538091851 >> 172.21.15.129:0/538091851 conn(0x7fb884077ba0 msgr2=0x7fb8840ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:04.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.918+0000 7fb88c4e1700 1 -- 172.21.15.129:0/538091851 shutdown_connections 2024-03-11T20:52:04.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:04.918+0000 7fb88c4e1700 1 -- 172.21.15.129:0/538091851 wait complete. 2024-03-11T20:52:04.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:04 smithi157 ceph-mon[33359]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-11T20:52:04.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:04.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:05.577 INFO:teuthology.orchestra.run.smithi129.stdout:34359738419 2024-03-11T20:52:05.577 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T20:52:06.028 INFO:teuthology.orchestra.run.smithi129.stdout:107374182434 2024-03-11T20:52:06.028 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-11T20:52:06.356 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:52:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:52:06.357 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[30184]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-11T20:52:06.357 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[30184]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-11T20:52:06.357 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:06.357 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:06.357 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:06.357 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:06.357 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:06.357 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:06.357 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[30184]: Adjusting osd_memory_target on smithi157 to 3760M 2024-03-11T20:52:06.358 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:06.358 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[42066]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-11T20:52:06.358 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[42066]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-11T20:52:06.358 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:06.358 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:06.358 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:06.358 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:06.358 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:06.358 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:06.358 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[42066]: Adjusting osd_memory_target on smithi157 to 3760M 2024-03-11T20:52:06.359 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:06 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:06 smithi157 ceph-mon[33359]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-11T20:52:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:06 smithi157 ceph-mon[33359]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-11T20:52:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:06 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:06 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:06 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:06 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:06 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:06 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:06 smithi157 ceph-mon[33359]: Adjusting osd_memory_target on smithi157 to 3760M 2024-03-11T20:52:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:06 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:06.664 INFO:teuthology.orchestra.run.smithi129.stdout:55834574893 2024-03-11T20:52:06.664 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T20:52:06.676 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:06.764 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:06.765 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:06.765 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:06.767 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:08.385 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:08 smithi129 ceph-mon[42066]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-11T20:52:08.386 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:08 smithi129 ceph-mon[30184]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-11T20:52:08.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:08 smithi157 ceph-mon[33359]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-11T20:52:08.709 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:10.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:10 smithi129 ceph-mon[30184]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T20:52:10.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:10 smithi129 ceph-mon[42066]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T20:52:10.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.480+0000 7fcd1db44700 1 -- 172.21.15.129:0/3312890491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd18107ee0 msgr2=0x7fcd181082c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:10.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.480+0000 7fcd1db44700 1 --2- 172.21.15.129:0/3312890491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd18107ee0 0x7fcd181082c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcd08009990 tx=0x7fcd08009ca0 comp rx=0 tx=0).stop 2024-03-11T20:52:10.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.480+0000 7fcd1db44700 1 -- 172.21.15.129:0/3312890491 shutdown_connections 2024-03-11T20:52:10.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.480+0000 7fcd1db44700 1 --2- 172.21.15.129:0/3312890491 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd181027f0 0x7fcd1810ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:10.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.480+0000 7fcd1db44700 1 --2- 172.21.15.129:0/3312890491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd18107ee0 0x7fcd181082c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:10.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.480+0000 7fcd1db44700 1 --2- 172.21.15.129:0/3312890491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd18101e30 0x7fcd181022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:10.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.480+0000 7fcd1db44700 1 -- 172.21.15.129:0/3312890491 >> 172.21.15.129:0/3312890491 conn(0x7fcd18078f10 msgr2=0x7fcd18079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:10.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.480+0000 7fcd1db44700 1 -- 172.21.15.129:0/3312890491 shutdown_connections 2024-03-11T20:52:10.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.480+0000 7fcd1db44700 1 -- 172.21.15.129:0/3312890491 wait complete. 2024-03-11T20:52:10.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.481+0000 7fcd1db44700 1 Processor -- start 2024-03-11T20:52:10.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.481+0000 7fcd1db44700 1 -- start start 2024-03-11T20:52:10.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.481+0000 7fcd1db44700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd18101e30 0x7fcd1819d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:10.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.481+0000 7fcd1db44700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd181027f0 0x7fcd1819dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:10.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.481+0000 7fcd1db44700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd18107ee0 0x7fcd18197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:10.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.481+0000 7fcd1db44700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd181115e0 con 0x7fcd18101e30 2024-03-11T20:52:10.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.481+0000 7fcd1db44700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd18111460 con 0x7fcd18107ee0 2024-03-11T20:52:10.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.481+0000 7fcd1db44700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd18111760 con 0x7fcd181027f0 2024-03-11T20:52:10.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.481+0000 7fcd177fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd18101e30 0x7fcd1819d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:10.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.481+0000 7fcd177fe700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd18101e30 0x7fcd1819d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35878/0 (socket says 172.21.15.129:35878) 2024-03-11T20:52:10.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.481+0000 7fcd177fe700 1 -- 172.21.15.129:0/3171783870 learned_addr learned my addr 172.21.15.129:0/3171783870 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:52:10.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.482+0000 7fcd16ffd700 1 --2- 172.21.15.129:0/3171783870 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd181027f0 0x7fcd1819dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:10.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.482+0000 7fcd17fff700 1 --2- 172.21.15.129:0/3171783870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd18107ee0 0x7fcd18197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:10.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.482+0000 7fcd177fe700 1 -- 172.21.15.129:0/3171783870 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd181027f0 msgr2=0x7fcd1819dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:10.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.482+0000 7fcd177fe700 1 --2- 172.21.15.129:0/3171783870 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd181027f0 0x7fcd1819dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:10.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.482+0000 7fcd177fe700 1 -- 172.21.15.129:0/3171783870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd18107ee0 msgr2=0x7fcd18197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:10.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.482+0000 7fcd177fe700 1 --2- 172.21.15.129:0/3171783870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd18107ee0 0x7fcd18197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:10.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.482+0000 7fcd177fe700 1 -- 172.21.15.129:0/3171783870 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd18198060 con 0x7fcd18101e30 2024-03-11T20:52:10.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.482+0000 7fcd16ffd700 1 --2- 172.21.15.129:0/3171783870 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd181027f0 0x7fcd1819dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:52:10.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.482+0000 7fcd177fe700 1 --2- 172.21.15.129:0/3171783870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd18101e30 0x7fcd1819d740 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcd080057c0 tx=0x7fcd080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:10.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.482+0000 7fcd14ff9700 1 -- 172.21.15.129:0/3171783870 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd08026070 con 0x7fcd18101e30 2024-03-11T20:52:10.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.482+0000 7fcd14ff9700 1 -- 172.21.15.129:0/3171783870 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fcd08021410 con 0x7fcd18101e30 2024-03-11T20:52:10.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.482+0000 7fcd1db44700 1 -- 172.21.15.129:0/3171783870 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd181982f0 con 0x7fcd18101e30 2024-03-11T20:52:10.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.482+0000 7fcd14ff9700 1 -- 172.21.15.129:0/3171783870 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd080173f0 con 0x7fcd18101e30 2024-03-11T20:52:10.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.482+0000 7fcd1db44700 1 -- 172.21.15.129:0/3171783870 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd181a4610 con 0x7fcd18101e30 2024-03-11T20:52:10.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.483+0000 7fcd1db44700 1 -- 172.21.15.129:0/3171783870 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd18068aa0 con 0x7fcd18101e30 2024-03-11T20:52:10.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.484+0000 7fcd14ff9700 1 -- 172.21.15.129:0/3171783870 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fcd08029070 con 0x7fcd18101e30 2024-03-11T20:52:10.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.484+0000 7fcd14ff9700 1 --2- 172.21.15.129:0/3171783870 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fcd0006de80 0x7fcd00070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:10.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.484+0000 7fcd16ffd700 1 --2- 172.21.15.129:0/3171783870 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fcd0006de80 0x7fcd00070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:10.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.484+0000 7fcd14ff9700 1 -- 172.21.15.129:0/3171783870 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7508+0+0 (secure 0 0 0) 0x7fcd0808efb0 con 0x7fcd18101e30 2024-03-11T20:52:10.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.484+0000 7fcd16ffd700 1 --2- 172.21.15.129:0/3171783870 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fcd0006de80 0x7fcd00070340 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcd18079710 tx=0x7fcd0c00b410 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:10.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:10 smithi157 ceph-mon[33359]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T20:52:10.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.489+0000 7fcd14ff9700 1 -- 172.21.15.129:0/3171783870 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fcd08013070 con 0x7fcd18101e30 2024-03-11T20:52:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.690+0000 7fcd1db44700 1 -- 172.21.15.129:0/3171783870 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fcd181a47e0 con 0x7fcd18101e30 2024-03-11T20:52:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.690+0000 7fcd14ff9700 1 -- 172.21.15.129:0/3171783870 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcd0805fee0 con 0x7fcd18101e30 2024-03-11T20:52:10.687 INFO:teuthology.orchestra.run.smithi129.stdout:77309411369 2024-03-11T20:52:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.693+0000 7fcd1db44700 1 -- 172.21.15.129:0/3171783870 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fcd0006de80 msgr2=0x7fcd00070340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.693+0000 7fcd1db44700 1 --2- 172.21.15.129:0/3171783870 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fcd0006de80 0x7fcd00070340 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcd18079710 tx=0x7fcd0c00b410 comp rx=0 tx=0).stop 2024-03-11T20:52:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.693+0000 7fcd1db44700 1 -- 172.21.15.129:0/3171783870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd18101e30 msgr2=0x7fcd1819d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.693+0000 7fcd1db44700 1 --2- 172.21.15.129:0/3171783870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd18101e30 0x7fcd1819d740 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcd080057c0 tx=0x7fcd080057f0 comp rx=0 tx=0).stop 2024-03-11T20:52:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.693+0000 7fcd1db44700 1 -- 172.21.15.129:0/3171783870 shutdown_connections 2024-03-11T20:52:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.693+0000 7fcd1db44700 1 --2- 172.21.15.129:0/3171783870 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fcd0006de80 0x7fcd00070340 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.693+0000 7fcd1db44700 1 --2- 172.21.15.129:0/3171783870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd18107ee0 0x7fcd18197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.693+0000 7fcd1db44700 1 --2- 172.21.15.129:0/3171783870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd18101e30 0x7fcd1819d740 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.693+0000 7fcd1db44700 1 --2- 172.21.15.129:0/3171783870 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcd181027f0 0x7fcd1819dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.693+0000 7fcd1db44700 1 -- 172.21.15.129:0/3171783870 >> 172.21.15.129:0/3171783870 conn(0x7fcd18078f10 msgr2=0x7fcd18108ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.694+0000 7fcd1db44700 1 -- 172.21.15.129:0/3171783870 shutdown_connections 2024-03-11T20:52:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:10.694+0000 7fcd1db44700 1 -- 172.21.15.129:0/3171783870 wait complete. 2024-03-11T20:52:11.364 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3171783870' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T20:52:11.365 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3171783870' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T20:52:11.387 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3171783870' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T20:52:12.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:12 smithi129 ceph-mon[30184]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T20:52:12.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:12 smithi129 ceph-mon[42066]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T20:52:12.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:12 smithi157 ceph-mon[33359]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T20:52:13.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.149+0000 7fcf1759e700 1 -- 172.21.15.129:0/407482318 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf18100e30 msgr2=0x7fcf18101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:13.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.149+0000 7fcf1759e700 1 --2- 172.21.15.129:0/407482318 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf18100e30 0x7fcf18101210 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcf08009920 tx=0x7fcf08009c30 comp rx=0 tx=0).stop 2024-03-11T20:52:13.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.150+0000 7fcf1759e700 1 -- 172.21.15.129:0/407482318 shutdown_connections 2024-03-11T20:52:13.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.150+0000 7fcf1759e700 1 --2- 172.21.15.129:0/407482318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf18105f10 0x7fcf1810dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.150+0000 7fcf1759e700 1 --2- 172.21.15.129:0/407482318 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf18100e30 0x7fcf18101210 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.150+0000 7fcf1759e700 1 --2- 172.21.15.129:0/407482318 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf181017e0 0x7fcf18105830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.150+0000 7fcf1759e700 1 -- 172.21.15.129:0/407482318 >> 172.21.15.129:0/407482318 conn(0x7fcf180fc6a0 msgr2=0x7fcf180feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:13.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.150+0000 7fcf1759e700 1 -- 172.21.15.129:0/407482318 shutdown_connections 2024-03-11T20:52:13.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.150+0000 7fcf1759e700 1 -- 172.21.15.129:0/407482318 wait complete. 2024-03-11T20:52:13.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.150+0000 7fcf1759e700 1 Processor -- start 2024-03-11T20:52:13.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.150+0000 7fcf1759e700 1 -- start start 2024-03-11T20:52:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.150+0000 7fcf1759e700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf18100e30 0x7fcf1819c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf1759e700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf181017e0 0x7fcf1819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf1759e700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf18105f10 0x7fcf18196760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf1759e700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf181103e0 con 0x7fcf18100e30 2024-03-11T20:52:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf1759e700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf18110260 con 0x7fcf181017e0 2024-03-11T20:52:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf1759e700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf18110560 con 0x7fcf18105f10 2024-03-11T20:52:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf1659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf18100e30 0x7fcf1819c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf1659c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf18100e30 0x7fcf1819c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35898/0 (socket says 172.21.15.129:35898) 2024-03-11T20:52:13.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf16d9d700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf18105f10 0x7fcf18196760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:13.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf1659c700 1 -- 172.21.15.129:0/769442968 learned_addr learned my addr 172.21.15.129:0/769442968 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:52:13.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf15d9b700 1 --2- 172.21.15.129:0/769442968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf181017e0 0x7fcf1819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:13.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf1659c700 1 -- 172.21.15.129:0/769442968 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf18105f10 msgr2=0x7fcf18196760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:13.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf1659c700 1 --2- 172.21.15.129:0/769442968 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf18105f10 0x7fcf18196760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf1659c700 1 -- 172.21.15.129:0/769442968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf181017e0 msgr2=0x7fcf1819cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:13.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf1659c700 1 --2- 172.21.15.129:0/769442968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf181017e0 0x7fcf1819cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf1659c700 1 -- 172.21.15.129:0/769442968 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf18197040 con 0x7fcf18100e30 2024-03-11T20:52:13.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf16d9d700 1 --2- 172.21.15.129:0/769442968 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf18105f10 0x7fcf18196760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:52:13.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf1659c700 1 --2- 172.21.15.129:0/769442968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf18100e30 0x7fcf1819c690 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcf08009230 tx=0x7fcf0800fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:13.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf0f7fe700 1 -- 172.21.15.129:0/769442968 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf08029070 con 0x7fcf18100e30 2024-03-11T20:52:13.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf0f7fe700 1 -- 172.21.15.129:0/769442968 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fcf0800bbc0 con 0x7fcf18100e30 2024-03-11T20:52:13.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf1759e700 1 -- 172.21.15.129:0/769442968 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf181972d0 con 0x7fcf18100e30 2024-03-11T20:52:13.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf0f7fe700 1 -- 172.21.15.129:0/769442968 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf08004930 con 0x7fcf18100e30 2024-03-11T20:52:13.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.151+0000 7fcf1759e700 1 -- 172.21.15.129:0/769442968 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1806a320 con 0x7fcf18100e30 2024-03-11T20:52:13.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.152+0000 7fcf1759e700 1 -- 172.21.15.129:0/769442968 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf18102310 con 0x7fcf18100e30 2024-03-11T20:52:13.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.153+0000 7fcf0f7fe700 1 -- 172.21.15.129:0/769442968 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fcf0802c070 con 0x7fcf18100e30 2024-03-11T20:52:13.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.153+0000 7fcf0f7fe700 1 --2- 172.21.15.129:0/769442968 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fcf0406de80 0x7fcf04070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:13.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.153+0000 7fcf0f7fe700 1 -- 172.21.15.129:0/769442968 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7508+0+0 (secure 0 0 0) 0x7fcf08090fb0 con 0x7fcf18100e30 2024-03-11T20:52:13.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.153+0000 7fcf15d9b700 1 --2- 172.21.15.129:0/769442968 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fcf0406de80 0x7fcf04070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:13.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.154+0000 7fcf15d9b700 1 --2- 172.21.15.129:0/769442968 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fcf0406de80 0x7fcf04070340 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcf00009750 tx=0x7fcf00009310 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:13.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.158+0000 7fcf0f7fe700 1 -- 172.21.15.129:0/769442968 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fcf08013030 con 0x7fcf18100e30 2024-03-11T20:52:13.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.358+0000 7fcf1759e700 1 -- 172.21.15.129:0/769442968 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fcf18197c90 con 0x7fcf18100e30 2024-03-11T20:52:13.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.358+0000 7fcf0f7fe700 1 -- 172.21.15.129:0/769442968 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fcf0805e490 con 0x7fcf18100e30 2024-03-11T20:52:13.354 INFO:teuthology.orchestra.run.smithi129.stdout:128849018912 2024-03-11T20:52:13.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.362+0000 7fcf1759e700 1 -- 172.21.15.129:0/769442968 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fcf0406de80 msgr2=0x7fcf04070340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:13.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.362+0000 7fcf1759e700 1 --2- 172.21.15.129:0/769442968 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fcf0406de80 0x7fcf04070340 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcf00009750 tx=0x7fcf00009310 comp rx=0 tx=0).stop 2024-03-11T20:52:13.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.362+0000 7fcf1759e700 1 -- 172.21.15.129:0/769442968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf18100e30 msgr2=0x7fcf1819c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:13.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.362+0000 7fcf1759e700 1 --2- 172.21.15.129:0/769442968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf18100e30 0x7fcf1819c690 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcf08009230 tx=0x7fcf0800fa70 comp rx=0 tx=0).stop 2024-03-11T20:52:13.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.362+0000 7fcf1759e700 1 -- 172.21.15.129:0/769442968 shutdown_connections 2024-03-11T20:52:13.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.362+0000 7fcf1759e700 1 --2- 172.21.15.129:0/769442968 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fcf0406de80 0x7fcf04070340 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.362+0000 7fcf1759e700 1 --2- 172.21.15.129:0/769442968 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fcf18105f10 0x7fcf18196760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.362+0000 7fcf1759e700 1 --2- 172.21.15.129:0/769442968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf18100e30 0x7fcf1819c690 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.362+0000 7fcf1759e700 1 --2- 172.21.15.129:0/769442968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf181017e0 0x7fcf1819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.362+0000 7fcf1759e700 1 -- 172.21.15.129:0/769442968 >> 172.21.15.129:0/769442968 conn(0x7fcf180fc6a0 msgr2=0x7fcf180fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:13.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.362+0000 7fcf1759e700 1 -- 172.21.15.129:0/769442968 shutdown_connections 2024-03-11T20:52:13.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.362+0000 7fcf1759e700 1 -- 172.21.15.129:0/769442968 wait complete. 2024-03-11T20:52:13.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.387+0000 7f3767a89700 1 -- 172.21.15.129:0/1692910568 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3760101740 msgr2=0x7f3760101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:13.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.387+0000 7f3767a89700 1 --2- 172.21.15.129:0/1692910568 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3760101740 0x7f3760101b20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f375c0099c0 tx=0x7f375c009cd0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.388+0000 7f3767a89700 1 -- 172.21.15.129:0/1692910568 shutdown_connections 2024-03-11T20:52:13.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.388+0000 7f3767a89700 1 --2- 172.21.15.129:0/1692910568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3760106060 0x7f376010dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.388+0000 7f3767a89700 1 --2- 172.21.15.129:0/1692910568 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3760101740 0x7f3760101b20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.388+0000 7f3767a89700 1 --2- 172.21.15.129:0/1692910568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37601020f0 0x7f3760105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.388+0000 7f3767a89700 1 -- 172.21.15.129:0/1692910568 >> 172.21.15.129:0/1692910568 conn(0x7f37600772e0 msgr2=0x7f37600fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:13.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.388+0000 7f3767a89700 1 -- 172.21.15.129:0/1692910568 shutdown_connections 2024-03-11T20:52:13.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.388+0000 7f3767a89700 1 -- 172.21.15.129:0/1692910568 wait complete. 2024-03-11T20:52:13.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.389+0000 7f3767a89700 1 Processor -- start 2024-03-11T20:52:13.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.389+0000 7f3767a89700 1 -- start start 2024-03-11T20:52:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.389+0000 7f3767a89700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3760101740 0x7f37601989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.389+0000 7f3767a89700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37601020f0 0x7f3760198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.389+0000 7f3767a89700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3760106060 0x7f37601a3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.389+0000 7f3767a89700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3760110570 con 0x7f37601020f0 2024-03-11T20:52:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.389+0000 7f3767a89700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37601103f0 con 0x7f3760101740 2024-03-11T20:52:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.389+0000 7f3767a89700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37601106f0 con 0x7f3760106060 2024-03-11T20:52:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.389+0000 7f3765024700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37601020f0 0x7f3760198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.389+0000 7f3766026700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3760106060 0x7f37601a3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.389+0000 7f3765024700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37601020f0 0x7f3760198f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35914/0 (socket says 172.21.15.129:35914) 2024-03-11T20:52:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.389+0000 7f3765024700 1 -- 172.21.15.129:0/3012898291 learned_addr learned my addr 172.21.15.129:0/3012898291 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:52:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.389+0000 7f3765825700 1 --2- 172.21.15.129:0/3012898291 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3760101740 0x7f37601989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.390+0000 7f3765024700 1 -- 172.21.15.129:0/3012898291 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3760106060 msgr2=0x7f37601a3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.390+0000 7f3765024700 1 --2- 172.21.15.129:0/3012898291 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3760106060 0x7f37601a3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.390+0000 7f3765024700 1 -- 172.21.15.129:0/3012898291 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3760101740 msgr2=0x7f37601989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:13.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.390+0000 7f3765024700 1 --2- 172.21.15.129:0/3012898291 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3760101740 0x7f37601989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.390+0000 7f3765024700 1 -- 172.21.15.129:0/3012898291 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37601a3f90 con 0x7f37601020f0 2024-03-11T20:52:13.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.390+0000 7f3765024700 1 --2- 172.21.15.129:0/3012898291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37601020f0 0x7f3760198f10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f375000c600 tx=0x7f375000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:13.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.390+0000 7f3756ffd700 1 -- 172.21.15.129:0/3012898291 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3750015d60 con 0x7f37601020f0 2024-03-11T20:52:13.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.390+0000 7f3767a89700 1 -- 172.21.15.129:0/3012898291 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3760196820 con 0x7f37601020f0 2024-03-11T20:52:13.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.390+0000 7f3756ffd700 1 -- 172.21.15.129:0/3012898291 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f37500044a0 con 0x7f37601020f0 2024-03-11T20:52:13.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.390+0000 7f3756ffd700 1 -- 172.21.15.129:0/3012898291 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375001e410 con 0x7f37601020f0 2024-03-11T20:52:13.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.390+0000 7f3767a89700 1 -- 172.21.15.129:0/3012898291 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3760196db0 con 0x7f37601020f0 2024-03-11T20:52:13.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.392+0000 7f3767a89700 1 -- 172.21.15.129:0/3012898291 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37601024f0 con 0x7f37601020f0 2024-03-11T20:52:13.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.392+0000 7f3756ffd700 1 -- 172.21.15.129:0/3012898291 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f3750003ee0 con 0x7f37601020f0 2024-03-11T20:52:13.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.393+0000 7f3756ffd700 1 --2- 172.21.15.129:0/3012898291 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f374c06de80 0x7f374c070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:13.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.393+0000 7f3756ffd700 1 -- 172.21.15.129:0/3012898291 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f3750096190 con 0x7f37601020f0 2024-03-11T20:52:13.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.393+0000 7f3765825700 1 --2- 172.21.15.129:0/3012898291 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f374c06de80 0x7f374c070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:13.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.394+0000 7f3765825700 1 --2- 172.21.15.129:0/3012898291 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f374c06de80 0x7f374c070340 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f375c003760 tx=0x7f375c005fb0 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:13.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.401+0000 7f3756ffd700 1 -- 172.21.15.129:0/3012898291 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f37500636f0 con 0x7f37601020f0 2024-03-11T20:52:13.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:52:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:52:13.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.600+0000 7f3767a89700 1 -- 172.21.15.129:0/3012898291 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f37601009c0 con 0x7f37601020f0 2024-03-11T20:52:13.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.601+0000 7f3756ffd700 1 -- 172.21.15.129:0/3012898291 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3750067140 con 0x7f37601020f0 2024-03-11T20:52:13.597 INFO:teuthology.orchestra.run.smithi129.stdout:231928234001 2024-03-11T20:52:13.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.603+0000 7f3767a89700 1 -- 172.21.15.129:0/3012898291 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f374c06de80 msgr2=0x7f374c070340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:13.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.603+0000 7f3767a89700 1 --2- 172.21.15.129:0/3012898291 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f374c06de80 0x7f374c070340 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f375c003760 tx=0x7f375c005fb0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.603+0000 7f3767a89700 1 -- 172.21.15.129:0/3012898291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37601020f0 msgr2=0x7f3760198f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:13.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.603+0000 7f3767a89700 1 --2- 172.21.15.129:0/3012898291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37601020f0 0x7f3760198f10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f375000c600 tx=0x7f375000c910 comp rx=0 tx=0).stop 2024-03-11T20:52:13.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.603+0000 7f3767a89700 1 -- 172.21.15.129:0/3012898291 shutdown_connections 2024-03-11T20:52:13.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.603+0000 7f3767a89700 1 --2- 172.21.15.129:0/3012898291 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f374c06de80 0x7f374c070340 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.603+0000 7f3767a89700 1 --2- 172.21.15.129:0/3012898291 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3760106060 0x7f37601a3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.603+0000 7f3767a89700 1 --2- 172.21.15.129:0/3012898291 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3760101740 0x7f37601989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.603+0000 7f3767a89700 1 --2- 172.21.15.129:0/3012898291 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37601020f0 0x7f3760198f10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:13.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.603+0000 7f3767a89700 1 -- 172.21.15.129:0/3012898291 >> 172.21.15.129:0/3012898291 conn(0x7f37600772e0 msgr2=0x7f37600fca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:13.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.603+0000 7f3767a89700 1 -- 172.21.15.129:0/3012898291 shutdown_connections 2024-03-11T20:52:13.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:13.603+0000 7f3767a89700 1 -- 172.21.15.129:0/3012898291 wait complete. 2024-03-11T20:52:14.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:14 smithi129 ceph-mon[30184]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T20:52:14.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:14 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:14.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/769442968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T20:52:14.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3012898291' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T20:52:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:14 smithi129 ceph-mon[42066]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T20:52:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:14 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/769442968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T20:52:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3012898291' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T20:52:14.360 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233999 got 231928234001 for osd.7 2024-03-11T20:52:14.361 DEBUG:teuthology.parallel:result is None 2024-03-11T20:52:14.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:14 smithi157 ceph-mon[33359]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-11T20:52:14.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:14 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:14.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/769442968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T20:52:14.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3012898291' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T20:52:15.047 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411369 for osd.2 2024-03-11T20:52:15.047 DEBUG:teuthology.parallel:result is None 2024-03-11T20:52:16.373 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:16 smithi129 ceph-mon[30184]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:52:16.373 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:16 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:16.373 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:16 smithi129 ceph-mon[42066]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:52:16.373 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:16 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:16.374 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:52:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:52:16.404 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018909 got 128849018912 for osd.4 2024-03-11T20:52:16.404 DEBUG:teuthology.parallel:result is None 2024-03-11T20:52:16.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:16 smithi157 ceph-mon[33359]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:52:16.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:16 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:16.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.581+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/2937972671 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a0102a40 msgr2=0x7fa7a0106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:16.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.581+0000 7fa7a5f4c700 1 --2- 172.21.15.129:0/2937972671 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a0102a40 0x7fa7a0106a80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa7940099c0 tx=0x7fa794009cd0 comp rx=0 tx=0).stop 2024-03-11T20:52:16.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.581+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/2937972671 shutdown_connections 2024-03-11T20:52:16.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.581+0000 7fa7a5f4c700 1 --2- 172.21.15.129:0/2937972671 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7a0107160 0x7fa7a010ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:16.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.581+0000 7fa7a5f4c700 1 --2- 172.21.15.129:0/2937972671 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7a0102090 0x7fa7a0102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:16.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.581+0000 7fa7a5f4c700 1 --2- 172.21.15.129:0/2937972671 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a0102a40 0x7fa7a0106a80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:16.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.581+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/2937972671 >> 172.21.15.129:0/2937972671 conn(0x7fa7a0078f40 msgr2=0x7fa7a0079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:16.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.581+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/2937972671 shutdown_connections 2024-03-11T20:52:16.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.581+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/2937972671 wait complete. 2024-03-11T20:52:16.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.582+0000 7fa7a5f4c700 1 Processor -- start 2024-03-11T20:52:16.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.582+0000 7fa7a5f4c700 1 -- start start 2024-03-11T20:52:16.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.582+0000 7fa7a5f4c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7a0102090 0x7fa7a00736c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:16.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.582+0000 7fa7a5f4c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a0102a40 0x7fa7a006c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:16.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.582+0000 7fa7a5f4c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7a0107160 0x7fa7a006cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:16.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.582+0000 7fa79effd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a0102a40 0x7fa7a006c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:16.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.582+0000 7fa79effd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a0102a40 0x7fa7a006c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40984/0 (socket says 172.21.15.129:40984) 2024-03-11T20:52:16.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.582+0000 7fa79effd700 1 -- 172.21.15.129:0/1248702040 learned_addr learned my addr 172.21.15.129:0/1248702040 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:52:16.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.583+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/1248702040 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a0111670 con 0x7fa7a0102a40 2024-03-11T20:52:16.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.583+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/1248702040 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a01114f0 con 0x7fa7a0107160 2024-03-11T20:52:16.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.583+0000 7fa79f7fe700 1 --2- 172.21.15.129:0/1248702040 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7a0102090 0x7fa7a00736c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:16.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.583+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/1248702040 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a01117f0 con 0x7fa7a0102090 2024-03-11T20:52:16.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.583+0000 7fa79f7fe700 1 -- 172.21.15.129:0/1248702040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7a0107160 msgr2=0x7fa7a006cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:16.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.583+0000 7fa79ffff700 1 --2- 172.21.15.129:0/1248702040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7a0107160 0x7fa7a006cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:16.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.583+0000 7fa79f7fe700 1 --2- 172.21.15.129:0/1248702040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7a0107160 0x7fa7a006cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:16.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.583+0000 7fa79f7fe700 1 -- 172.21.15.129:0/1248702040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a0102a40 msgr2=0x7fa7a006c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:16.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.583+0000 7fa79f7fe700 1 --2- 172.21.15.129:0/1248702040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a0102a40 0x7fa7a006c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:16.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.583+0000 7fa79f7fe700 1 -- 172.21.15.129:0/1248702040 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a006d410 con 0x7fa7a0102090 2024-03-11T20:52:16.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.583+0000 7fa79effd700 1 --2- 172.21.15.129:0/1248702040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a0102a40 0x7fa7a006c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:52:16.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.583+0000 7fa79ffff700 1 --2- 172.21.15.129:0/1248702040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7a0107160 0x7fa7a006cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T20:52:16.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.583+0000 7fa79f7fe700 1 --2- 172.21.15.129:0/1248702040 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7a0102090 0x7fa7a00736c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa79000d950 tx=0x7fa79000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:16.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.584+0000 7fa79cff9700 1 -- 172.21.15.129:0/1248702040 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa790017070 con 0x7fa7a0102090 2024-03-11T20:52:16.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.584+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/1248702040 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a00711f0 con 0x7fa7a0102090 2024-03-11T20:52:16.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.584+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/1248702040 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a0071750 con 0x7fa7a0102090 2024-03-11T20:52:16.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.584+0000 7fa79cff9700 1 -- 172.21.15.129:0/1248702040 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa790009db0 con 0x7fa7a0102090 2024-03-11T20:52:16.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.584+0000 7fa79cff9700 1 -- 172.21.15.129:0/1248702040 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa790010ca0 con 0x7fa7a0102090 2024-03-11T20:52:16.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.585+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/1248702040 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa78c005120 con 0x7fa7a0102090 2024-03-11T20:52:16.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.586+0000 7fa79cff9700 1 -- 172.21.15.129:0/1248702040 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fa79001a070 con 0x7fa7a0102090 2024-03-11T20:52:16.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.586+0000 7fa79cff9700 1 --2- 172.21.15.129:0/1248702040 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fa78806de80 0x7fa788070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:16.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.587+0000 7fa79cff9700 1 -- 172.21.15.129:0/1248702040 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7508+0+0 (secure 0 0 0) 0x7fa79008db10 con 0x7fa7a0102090 2024-03-11T20:52:16.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.587+0000 7fa79effd700 1 --2- 172.21.15.129:0/1248702040 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fa78806de80 0x7fa788070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:16.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.587+0000 7fa79effd700 1 --2- 172.21.15.129:0/1248702040 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fa78806de80 0x7fa788070340 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa794009990 tx=0x7fa794005960 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:16.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.592+0000 7fa79cff9700 1 -- 172.21.15.129:0/1248702040 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fa79005afd0 con 0x7fa7a0102090 2024-03-11T20:52:16.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.803+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/1248702040 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa78c005410 con 0x7fa7a0102090 2024-03-11T20:52:16.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.803+0000 7fa79cff9700 1 -- 172.21.15.129:0/1248702040 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa79005ea20 con 0x7fa7a0102090 2024-03-11T20:52:16.799 INFO:teuthology.orchestra.run.smithi129.stdout:197568495639 2024-03-11T20:52:16.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.805+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/1248702040 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fa78806de80 msgr2=0x7fa788070340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:16.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.805+0000 7fa7a5f4c700 1 --2- 172.21.15.129:0/1248702040 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fa78806de80 0x7fa788070340 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa794009990 tx=0x7fa794005960 comp rx=0 tx=0).stop 2024-03-11T20:52:16.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.805+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/1248702040 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7a0102090 msgr2=0x7fa7a00736c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:16.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.805+0000 7fa7a5f4c700 1 --2- 172.21.15.129:0/1248702040 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7a0102090 0x7fa7a00736c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa79000d950 tx=0x7fa79000dc60 comp rx=0 tx=0).stop 2024-03-11T20:52:16.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.805+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/1248702040 shutdown_connections 2024-03-11T20:52:16.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.805+0000 7fa7a5f4c700 1 --2- 172.21.15.129:0/1248702040 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fa78806de80 0x7fa788070340 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:16.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.805+0000 7fa7a5f4c700 1 --2- 172.21.15.129:0/1248702040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7a0107160 0x7fa7a006cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:16.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.805+0000 7fa7a5f4c700 1 --2- 172.21.15.129:0/1248702040 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fa7a0102090 0x7fa7a00736c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:16.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.805+0000 7fa7a5f4c700 1 --2- 172.21.15.129:0/1248702040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a0102a40 0x7fa7a006c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:16.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.805+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/1248702040 >> 172.21.15.129:0/1248702040 conn(0x7fa7a0078f40 msgr2=0x7fa7a01001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:16.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.805+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/1248702040 shutdown_connections 2024-03-11T20:52:16.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:16.805+0000 7fa7a5f4c700 1 -- 172.21.15.129:0/1248702040 wait complete. 2024-03-11T20:52:17.397 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1248702040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T20:52:17.398 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1248702040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T20:52:17.429 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495636 got 197568495639 for osd.6 2024-03-11T20:52:17.429 DEBUG:teuthology.parallel:result is None 2024-03-11T20:52:17.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1248702040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T20:52:17.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.613+0000 7fb6af15c700 1 -- 172.21.15.129:0/3702054738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a81071b0 msgr2=0x7fb6a810ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:17.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.613+0000 7fb6af15c700 1 --2- 172.21.15.129:0/3702054738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a81071b0 0x7fb6a810ed10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0009fc0 tx=0x7fb6a000b330 comp rx=0 tx=0).stop 2024-03-11T20:52:17.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.613+0000 7fb6af15c700 1 -- 172.21.15.129:0/3702054738 shutdown_connections 2024-03-11T20:52:17.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.613+0000 7fb6af15c700 1 --2- 172.21.15.129:0/3702054738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a81071b0 0x7fb6a810ed10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:17.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.613+0000 7fb6af15c700 1 --2- 172.21.15.129:0/3702054738 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6a8100590 0x7fb6a8100970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:17.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.613+0000 7fb6af15c700 1 --2- 172.21.15.129:0/3702054738 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb6a8100f40 0x7fb6a8106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:17.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.613+0000 7fb6af15c700 1 -- 172.21.15.129:0/3702054738 >> 172.21.15.129:0/3702054738 conn(0x7fb6a80772e0 msgr2=0x7fb6a80fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:17.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.613+0000 7fb6af15c700 1 -- 172.21.15.129:0/3702054738 shutdown_connections 2024-03-11T20:52:17.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.613+0000 7fb6af15c700 1 -- 172.21.15.129:0/3702054738 wait complete. 2024-03-11T20:52:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.614+0000 7fb6af15c700 1 Processor -- start 2024-03-11T20:52:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.614+0000 7fb6af15c700 1 -- start start 2024-03-11T20:52:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.614+0000 7fb6af15c700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb6a8100590 0x7fb6a819d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.614+0000 7fb6af15c700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a8100f40 0x7fb6a819dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.614+0000 7fb6af15c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6a81071b0 0x7fb6a8197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.614+0000 7fb6af15c700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a81115d0 con 0x7fb6a8100f40 2024-03-11T20:52:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.614+0000 7fb6af15c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a8111450 con 0x7fb6a81071b0 2024-03-11T20:52:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.614+0000 7fb6af15c700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a8111750 con 0x7fb6a8100590 2024-03-11T20:52:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.614+0000 7fb6acef8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb6a8100590 0x7fb6a819d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.614+0000 7fb6a7fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a8100f40 0x7fb6a819dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.614+0000 7fb6acef8700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb6a8100590 0x7fb6a819d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:40280/0 (socket says 172.21.15.129:40280) 2024-03-11T20:52:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.614+0000 7fb6a7fff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a8100f40 0x7fb6a819dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40990/0 (socket says 172.21.15.129:40990) 2024-03-11T20:52:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.614+0000 7fb6acef8700 1 -- 172.21.15.129:0/2196608322 learned_addr learned my addr 172.21.15.129:0/2196608322 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:52:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.615+0000 7fb6ad6f9700 1 --2- 172.21.15.129:0/2196608322 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6a81071b0 0x7fb6a8197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.615+0000 7fb6acef8700 1 -- 172.21.15.129:0/2196608322 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6a81071b0 msgr2=0x7fb6a8197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.615+0000 7fb6acef8700 1 --2- 172.21.15.129:0/2196608322 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6a81071b0 0x7fb6a8197840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.615+0000 7fb6acef8700 1 -- 172.21.15.129:0/2196608322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a8100f40 msgr2=0x7fb6a819dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.615+0000 7fb6acef8700 1 --2- 172.21.15.129:0/2196608322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a8100f40 0x7fb6a819dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.615+0000 7fb6acef8700 1 -- 172.21.15.129:0/2196608322 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a8198020 con 0x7fb6a8100590 2024-03-11T20:52:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.615+0000 7fb6a7fff700 1 --2- 172.21.15.129:0/2196608322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a8100f40 0x7fb6a819dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:52:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.615+0000 7fb6ad6f9700 1 --2- 172.21.15.129:0/2196608322 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6a81071b0 0x7fb6a8197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T20:52:17.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.615+0000 7fb6acef8700 1 --2- 172.21.15.129:0/2196608322 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb6a8100590 0x7fb6a819d6c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00b530 tx=0x7fb69c00b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:17.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.615+0000 7fb6a5ffb700 1 -- 172.21.15.129:0/2196608322 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c015020 con 0x7fb6a8100590 2024-03-11T20:52:17.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.615+0000 7fb6af15c700 1 -- 172.21.15.129:0/2196608322 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a8198310 con 0x7fb6a8100590 2024-03-11T20:52:17.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.615+0000 7fb6a5ffb700 1 -- 172.21.15.129:0/2196608322 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fb69c011cb0 con 0x7fb6a8100590 2024-03-11T20:52:17.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.615+0000 7fb6a5ffb700 1 -- 172.21.15.129:0/2196608322 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69c010d50 con 0x7fb6a8100590 2024-03-11T20:52:17.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.615+0000 7fb6af15c700 1 -- 172.21.15.129:0/2196608322 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a81a45a0 con 0x7fb6a8100590 2024-03-11T20:52:17.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.616+0000 7fb6af15c700 1 -- 172.21.15.129:0/2196608322 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a8103650 con 0x7fb6a8100590 2024-03-11T20:52:17.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.617+0000 7fb6a5ffb700 1 -- 172.21.15.129:0/2196608322 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fb69c018020 con 0x7fb6a8100590 2024-03-11T20:52:17.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.617+0000 7fb6a5ffb700 1 --2- 172.21.15.129:0/2196608322 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fb69806de80 0x7fb698070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:17.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.617+0000 7fb6a5ffb700 1 -- 172.21.15.129:0/2196608322 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7508+0+0 (secure 0 0 0) 0x7fb69c08dab0 con 0x7fb6a8100590 2024-03-11T20:52:17.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.617+0000 7fb6a7fff700 1 --2- 172.21.15.129:0/2196608322 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fb69806de80 0x7fb698070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:17.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.618+0000 7fb6a7fff700 1 --2- 172.21.15.129:0/2196608322 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fb69806de80 0x7fb698070340 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb694009750 tx=0x7fb694009310 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:17.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.622+0000 7fb6a5ffb700 1 -- 172.21.15.129:0/2196608322 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fb69c01f020 con 0x7fb6a8100590 2024-03-11T20:52:17.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.837+0000 7fb6af15c700 1 -- 172.21.15.129:0/2196608322 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb6a8198d70 con 0x7fb6a8100590 2024-03-11T20:52:17.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.838+0000 7fb6a5ffb700 1 -- 172.21.15.129:0/2196608322 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb69c05af90 con 0x7fb6a8100590 2024-03-11T20:52:17.834 INFO:teuthology.orchestra.run.smithi129.stdout:163208757276 2024-03-11T20:52:17.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.840+0000 7fb6af15c700 1 -- 172.21.15.129:0/2196608322 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fb69806de80 msgr2=0x7fb698070340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:17.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.840+0000 7fb6af15c700 1 --2- 172.21.15.129:0/2196608322 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fb69806de80 0x7fb698070340 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb694009750 tx=0x7fb694009310 comp rx=0 tx=0).stop 2024-03-11T20:52:17.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.840+0000 7fb6af15c700 1 -- 172.21.15.129:0/2196608322 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb6a8100590 msgr2=0x7fb6a819d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:17.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.840+0000 7fb6af15c700 1 --2- 172.21.15.129:0/2196608322 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb6a8100590 0x7fb6a819d6c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00b530 tx=0x7fb69c00b840 comp rx=0 tx=0).stop 2024-03-11T20:52:17.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.840+0000 7fb6af15c700 1 -- 172.21.15.129:0/2196608322 shutdown_connections 2024-03-11T20:52:17.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.840+0000 7fb6af15c700 1 --2- 172.21.15.129:0/2196608322 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fb69806de80 0x7fb698070340 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:17.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.840+0000 7fb6af15c700 1 --2- 172.21.15.129:0/2196608322 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6a81071b0 0x7fb6a8197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:17.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.840+0000 7fb6af15c700 1 --2- 172.21.15.129:0/2196608322 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fb6a8100590 0x7fb6a819d6c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:17.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.840+0000 7fb6af15c700 1 --2- 172.21.15.129:0/2196608322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a8100f40 0x7fb6a819dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:17.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.840+0000 7fb6af15c700 1 -- 172.21.15.129:0/2196608322 >> 172.21.15.129:0/2196608322 conn(0x7fb6a80772e0 msgr2=0x7fb6a8104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:17.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.840+0000 7fb6af15c700 1 -- 172.21.15.129:0/2196608322 shutdown_connections 2024-03-11T20:52:17.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:17.840+0000 7fb6af15c700 1 -- 172.21.15.129:0/2196608322 wait complete. 2024-03-11T20:52:18.425 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:18.425 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:18.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:18 smithi129 ceph-mon[30184]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:52:18.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2196608322' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T20:52:18.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:18 smithi129 ceph-mon[42066]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:52:18.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2196608322' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T20:52:18.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:18 smithi157 ceph-mon[33359]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:52:18.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2196608322' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T20:52:19.524 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757272 got 163208757276 for osd.5 2024-03-11T20:52:19.524 DEBUG:teuthology.parallel:result is None 2024-03-11T20:52:20.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:20 smithi129 ceph-mon[30184]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:52:20.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:20.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:20.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:20 smithi129 ceph-mon[42066]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:52:20.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:20.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:20.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:20 smithi157 ceph-mon[33359]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T20:52:20.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:20.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:21.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.137+0000 7f14d2899700 1 -- 172.21.15.129:0/3339465465 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc1018d0 msgr2=0x7f14cc105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:21.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.137+0000 7f14d2899700 1 --2- 172.21.15.129:0/3339465465 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc1018d0 0x7f14cc105900 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f14c00098f0 tx=0x7f14c0009c00 comp rx=0 tx=0).stop 2024-03-11T20:52:21.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.137+0000 7f14d2899700 1 -- 172.21.15.129:0/3339465465 shutdown_connections 2024-03-11T20:52:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.137+0000 7f14d2899700 1 --2- 172.21.15.129:0/3339465465 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14cc105fe0 0x7f14cc10dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.137+0000 7f14d2899700 1 --2- 172.21.15.129:0/3339465465 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14cc100f20 0x7f14cc101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.137+0000 7f14d2899700 1 --2- 172.21.15.129:0/3339465465 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc1018d0 0x7f14cc105900 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.137+0000 7f14d2899700 1 -- 172.21.15.129:0/3339465465 >> 172.21.15.129:0/3339465465 conn(0x7f14cc077c80 msgr2=0x7f14cc078090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.137+0000 7f14d2899700 1 -- 172.21.15.129:0/3339465465 shutdown_connections 2024-03-11T20:52:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.137+0000 7f14d2899700 1 -- 172.21.15.129:0/3339465465 wait complete. 2024-03-11T20:52:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14d2899700 1 Processor -- start 2024-03-11T20:52:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14d2899700 1 -- start start 2024-03-11T20:52:21.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14d2899700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc100f20 0x7f14cc19d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:21.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14d2899700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14cc1018d0 0x7f14cc19dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:21.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14d2899700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14cc105fe0 0x7f14cc1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:21.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14d2899700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc1104d0 con 0x7f14cc100f20 2024-03-11T20:52:21.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14d2899700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc110350 con 0x7f14cc105fe0 2024-03-11T20:52:21.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14d2899700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc110650 con 0x7f14cc1018d0 2024-03-11T20:52:21.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14cb7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14cc1018d0 0x7f14cc19dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:21.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14cbfff700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc100f20 0x7f14cc19d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:21.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14cb7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14cc1018d0 0x7f14cc19dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:40296/0 (socket says 172.21.15.129:40296) 2024-03-11T20:52:21.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14cb7fe700 1 -- 172.21.15.129:0/1327338538 learned_addr learned my addr 172.21.15.129:0/1327338538 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:52:21.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14d0e36700 1 --2- 172.21.15.129:0/1327338538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14cc105fe0 0x7f14cc1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:21.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14cb7fe700 1 -- 172.21.15.129:0/1327338538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14cc105fe0 msgr2=0x7f14cc1979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:21.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14cb7fe700 1 --2- 172.21.15.129:0/1327338538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14cc105fe0 0x7f14cc1979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:21.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14cb7fe700 1 -- 172.21.15.129:0/1327338538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc100f20 msgr2=0x7f14cc19d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:21.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14cb7fe700 1 --2- 172.21.15.129:0/1327338538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc100f20 0x7f14cc19d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:21.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14cb7fe700 1 -- 172.21.15.129:0/1327338538 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14cc198100 con 0x7f14cc1018d0 2024-03-11T20:52:21.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14cbfff700 1 --2- 172.21.15.129:0/1327338538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc100f20 0x7f14cc19d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:52:21.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.138+0000 7f14d0e36700 1 --2- 172.21.15.129:0/1327338538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14cc105fe0 0x7f14cc1979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T20:52:21.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.139+0000 7f14cb7fe700 1 --2- 172.21.15.129:0/1327338538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14cc1018d0 0x7f14cc19dcd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f14c0009230 tx=0x7f14c000fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:21.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.139+0000 7f14c97fa700 1 -- 172.21.15.129:0/1327338538 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c000b6e0 con 0x7f14cc1018d0 2024-03-11T20:52:21.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.139+0000 7f14c97fa700 1 -- 172.21.15.129:0/1327338538 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f14c0005040 con 0x7f14cc1018d0 2024-03-11T20:52:21.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.139+0000 7f14d2899700 1 -- 172.21.15.129:0/1327338538 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14cc198390 con 0x7f14cc1018d0 2024-03-11T20:52:21.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.139+0000 7f14c97fa700 1 -- 172.21.15.129:0/1327338538 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c0003990 con 0x7f14cc1018d0 2024-03-11T20:52:21.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.139+0000 7f14d2899700 1 -- 172.21.15.129:0/1327338538 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14cc1a4660 con 0x7f14cc1018d0 2024-03-11T20:52:21.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.140+0000 7f14d2899700 1 -- 172.21.15.129:0/1327338538 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14cc068aa0 con 0x7f14cc1018d0 2024-03-11T20:52:21.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.140+0000 7f14c97fa700 1 -- 172.21.15.129:0/1327338538 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f14c000b880 con 0x7f14cc1018d0 2024-03-11T20:52:21.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.141+0000 7f14c97fa700 1 --2- 172.21.15.129:0/1327338538 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f14b406de80 0x7f14b4070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:21.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.141+0000 7f14c97fa700 1 -- 172.21.15.129:0/1327338538 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f14c008f370 con 0x7f14cc1018d0 2024-03-11T20:52:21.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.141+0000 7f14cbfff700 1 --2- 172.21.15.129:0/1327338538 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f14b406de80 0x7f14b4070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:21.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.141+0000 7f14cbfff700 1 --2- 172.21.15.129:0/1327338538 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f14b406de80 0x7f14b4070340 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f14cc1055b0 tx=0x7f14bc00b3c0 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:21.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.145+0000 7f14c97fa700 1 -- 172.21.15.129:0/1327338538 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f14c0013030 con 0x7f14cc1018d0 2024-03-11T20:52:21.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.351+0000 7f14d2899700 1 -- 172.21.15.129:0/1327338538 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f14cc1a4830 con 0x7f14cc1018d0 2024-03-11T20:52:21.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.352+0000 7f14c97fa700 1 -- 172.21.15.129:0/1327338538 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f14c005c8d0 con 0x7f14cc1018d0 2024-03-11T20:52:21.348 INFO:teuthology.orchestra.run.smithi129.stdout:34359738423 2024-03-11T20:52:21.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.355+0000 7f14d2899700 1 -- 172.21.15.129:0/1327338538 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f14b406de80 msgr2=0x7f14b4070340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:21.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.355+0000 7f14d2899700 1 --2- 172.21.15.129:0/1327338538 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f14b406de80 0x7f14b4070340 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f14cc1055b0 tx=0x7f14bc00b3c0 comp rx=0 tx=0).stop 2024-03-11T20:52:21.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.355+0000 7f14d2899700 1 -- 172.21.15.129:0/1327338538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14cc1018d0 msgr2=0x7f14cc19dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:21.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.355+0000 7f14d2899700 1 --2- 172.21.15.129:0/1327338538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14cc1018d0 0x7f14cc19dcd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f14c0009230 tx=0x7f14c000fa00 comp rx=0 tx=0).stop 2024-03-11T20:52:21.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.355+0000 7f14d2899700 1 -- 172.21.15.129:0/1327338538 shutdown_connections 2024-03-11T20:52:21.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.355+0000 7f14d2899700 1 --2- 172.21.15.129:0/1327338538 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f14b406de80 0x7f14b4070340 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:21.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.355+0000 7f14d2899700 1 --2- 172.21.15.129:0/1327338538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14cc105fe0 0x7f14cc1979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:21.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.355+0000 7f14d2899700 1 --2- 172.21.15.129:0/1327338538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14cc100f20 0x7f14cc19d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:21.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.355+0000 7f14d2899700 1 --2- 172.21.15.129:0/1327338538 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14cc1018d0 0x7f14cc19dcd0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:21.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.355+0000 7f14d2899700 1 -- 172.21.15.129:0/1327338538 >> 172.21.15.129:0/1327338538 conn(0x7f14cc077c80 msgr2=0x7f14cc06a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:21.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.355+0000 7f14d2899700 1 -- 172.21.15.129:0/1327338538 shutdown_connections 2024-03-11T20:52:21.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:21.355+0000 7f14d2899700 1 -- 172.21.15.129:0/1327338538 wait complete. 2024-03-11T20:52:21.598 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1327338538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T20:52:21.598 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1327338538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T20:52:21.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1327338538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T20:52:22.695 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:22 smithi129 ceph-mon[30184]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:22.695 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:22 smithi129 ceph-mon[42066]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:22.771 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738419 got 34359738423 for osd.0 2024-03-11T20:52:22.771 DEBUG:teuthology.parallel:result is None 2024-03-11T20:52:22.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:22 smithi157 ceph-mon[33359]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.342+0000 7f0223db7700 1 -- 172.21.15.129:0/1780543554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f021c101920 msgr2=0x7f021c1058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.342+0000 7f0223db7700 1 --2- 172.21.15.129:0/1780543554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f021c101920 0x7f021c1058e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f020c009920 tx=0x7f020c009c30 comp rx=0 tx=0).stop 2024-03-11T20:52:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.343+0000 7f0223db7700 1 -- 172.21.15.129:0/1780543554 shutdown_connections 2024-03-11T20:52:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.343+0000 7f0223db7700 1 --2- 172.21.15.129:0/1780543554 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f021c105fc0 0x7f021c10db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.343+0000 7f0223db7700 1 --2- 172.21.15.129:0/1780543554 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f021c100f70 0x7f021c101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.343+0000 7f0223db7700 1 --2- 172.21.15.129:0/1780543554 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f021c101920 0x7f021c1058e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.343+0000 7f0223db7700 1 -- 172.21.15.129:0/1780543554 >> 172.21.15.129:0/1780543554 conn(0x7f021c076d30 msgr2=0x7f021c077140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.343+0000 7f0223db7700 1 -- 172.21.15.129:0/1780543554 shutdown_connections 2024-03-11T20:52:23.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.343+0000 7f0223db7700 1 -- 172.21.15.129:0/1780543554 wait complete. 2024-03-11T20:52:23.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.343+0000 7f0223db7700 1 Processor -- start 2024-03-11T20:52:23.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0223db7700 1 -- start start 2024-03-11T20:52:23.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0223db7700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f021c100f70 0x7f021c1a3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:23.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0223db7700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f021c101920 0x7f021c1a4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:23.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0223db7700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f021c105fc0 0x7f021c19e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:23.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0223db7700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c1104d0 con 0x7f021c101920 2024-03-11T20:52:23.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0223db7700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c110350 con 0x7f021c100f70 2024-03-11T20:52:23.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0223db7700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021c110650 con 0x7f021c105fc0 2024-03-11T20:52:23.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0221352700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f021c101920 0x7f021c1a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:23.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0222354700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f021c105fc0 0x7f021c19e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:23.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0221352700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f021c101920 0x7f021c1a4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41032/0 (socket says 172.21.15.129:41032) 2024-03-11T20:52:23.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0222354700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f021c105fc0 0x7f021c19e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:40312/0 (socket says 172.21.15.129:40312) 2024-03-11T20:52:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0221352700 1 -- 172.21.15.129:0/4075675345 learned_addr learned my addr 172.21.15.129:0/4075675345 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:52:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0221b53700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f021c100f70 0x7f021c1a3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0222354700 1 -- 172.21.15.129:0/4075675345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f021c100f70 msgr2=0x7f021c1a3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0222354700 1 --2- 172.21.15.129:0/4075675345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f021c100f70 0x7f021c1a3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0222354700 1 -- 172.21.15.129:0/4075675345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f021c101920 msgr2=0x7f021c1a4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0222354700 1 --2- 172.21.15.129:0/4075675345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f021c101920 0x7f021c1a4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0222354700 1 -- 172.21.15.129:0/4075675345 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c19e920 con 0x7f021c105fc0 2024-03-11T20:52:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0221352700 1 --2- 172.21.15.129:0/4075675345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f021c101920 0x7f021c1a4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:52:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.344+0000 7f0222354700 1 --2- 172.21.15.129:0/4075675345 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f021c105fc0 0x7f021c19e150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0214009cd0 tx=0x7f021400c620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.345+0000 7f0212ffd700 1 -- 172.21.15.129:0/4075675345 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021400cd10 con 0x7f021c105fc0 2024-03-11T20:52:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.345+0000 7f0223db7700 1 -- 172.21.15.129:0/4075675345 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021c19ebb0 con 0x7f021c105fc0 2024-03-11T20:52:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.345+0000 7f0212ffd700 1 -- 172.21.15.129:0/4075675345 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0214003a00 con 0x7f021c105fc0 2024-03-11T20:52:23.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.345+0000 7f0212ffd700 1 -- 172.21.15.129:0/4075675345 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0214004350 con 0x7f021c105fc0 2024-03-11T20:52:23.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.345+0000 7f0223db7700 1 -- 172.21.15.129:0/4075675345 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f021c1aac60 con 0x7f021c105fc0 2024-03-11T20:52:23.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.346+0000 7f0223db7700 1 -- 172.21.15.129:0/4075675345 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f021c068aa0 con 0x7f021c105fc0 2024-03-11T20:52:23.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.346+0000 7f0212ffd700 1 -- 172.21.15.129:0/4075675345 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f021401a020 con 0x7f021c105fc0 2024-03-11T20:52:23.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.347+0000 7f0212ffd700 1 --2- 172.21.15.129:0/4075675345 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f020806de80 0x7f0208070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:23.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.347+0000 7f0212ffd700 1 -- 172.21.15.129:0/4075675345 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f021408db20 con 0x7f021c105fc0 2024-03-11T20:52:23.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.347+0000 7f0221b53700 1 --2- 172.21.15.129:0/4075675345 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f020806de80 0x7f0208070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:23.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.347+0000 7f0221b53700 1 --2- 172.21.15.129:0/4075675345 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f020806de80 0x7f0208070340 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0218005950 tx=0x7f021800b410 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:23.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.352+0000 7f0212ffd700 1 -- 172.21.15.129:0/4075675345 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f0214010040 con 0x7f021c105fc0 2024-03-11T20:52:23.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:52:23.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.566+0000 7f0223db7700 1 -- 172.21.15.129:0/4075675345 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f021c1aae90 con 0x7f021c105fc0 2024-03-11T20:52:23.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.567+0000 7f0212ffd700 1 -- 172.21.15.129:0/4075675345 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f021405b000 con 0x7f021c105fc0 2024-03-11T20:52:23.563 INFO:teuthology.orchestra.run.smithi129.stdout:55834574896 2024-03-11T20:52:23.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.569+0000 7f0223db7700 1 -- 172.21.15.129:0/4075675345 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f020806de80 msgr2=0x7f0208070340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:23.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.569+0000 7f0223db7700 1 --2- 172.21.15.129:0/4075675345 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f020806de80 0x7f0208070340 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0218005950 tx=0x7f021800b410 comp rx=0 tx=0).stop 2024-03-11T20:52:23.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.569+0000 7f0223db7700 1 -- 172.21.15.129:0/4075675345 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f021c105fc0 msgr2=0x7f021c19e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:23.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.569+0000 7f0223db7700 1 --2- 172.21.15.129:0/4075675345 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f021c105fc0 0x7f021c19e150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0214009cd0 tx=0x7f021400c620 comp rx=0 tx=0).stop 2024-03-11T20:52:23.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.569+0000 7f0223db7700 1 -- 172.21.15.129:0/4075675345 shutdown_connections 2024-03-11T20:52:23.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.569+0000 7f0223db7700 1 --2- 172.21.15.129:0/4075675345 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f020806de80 0x7f0208070340 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:23.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.569+0000 7f0223db7700 1 --2- 172.21.15.129:0/4075675345 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f021c105fc0 0x7f021c19e150 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:23.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.569+0000 7f0223db7700 1 --2- 172.21.15.129:0/4075675345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f021c100f70 0x7f021c1a3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:23.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.569+0000 7f0223db7700 1 --2- 172.21.15.129:0/4075675345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f021c101920 0x7f021c1a4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:23.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.569+0000 7f0223db7700 1 -- 172.21.15.129:0/4075675345 >> 172.21.15.129:0/4075675345 conn(0x7f021c076d30 msgr2=0x7f021c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:23.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.569+0000 7f0223db7700 1 -- 172.21.15.129:0/4075675345 shutdown_connections 2024-03-11T20:52:23.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:23.569+0000 7f0223db7700 1 -- 172.21.15.129:0/4075675345 wait complete. 2024-03-11T20:52:23.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:23.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: Adjusting osd_memory_target on smithi129 to 3760M 2024-03-11T20:52:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:52:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:52:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-11T20:52:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: Updating smithi157:/etc/ceph/ceph.conf 2024-03-11T20:52:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: Updating smithi129:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:52:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: Updating smithi157:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:52:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:23.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:52:23.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: Updating smithi157:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:52:23.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4075675345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T20:52:23.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: Updating smithi157:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:52:23.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[30184]: Updating smithi129:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:52:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: Adjusting osd_memory_target on smithi129 to 3760M 2024-03-11T20:52:23.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:23.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:52:23.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:52:23.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-11T20:52:23.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: Updating smithi157:/etc/ceph/ceph.conf 2024-03-11T20:52:23.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: Updating smithi129:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:52:23.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: Updating smithi157:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:52:23.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:23.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:52:23.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: Updating smithi157:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:52:23.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4075675345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T20:52:23.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: Updating smithi157:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:52:23.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:23 smithi129 ceph-mon[42066]: Updating smithi129:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:52:24.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:24.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:24.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T20:52:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: Adjusting osd_memory_target on smithi129 to 3760M 2024-03-11T20:52:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:52:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:52:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: Updating smithi129:/etc/ceph/ceph.conf 2024-03-11T20:52:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: Updating smithi157:/etc/ceph/ceph.conf 2024-03-11T20:52:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: Updating smithi129:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:52:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: Updating smithi157:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T20:52:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:52:24.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: Updating smithi157:/etc/ceph/ceph.client.admin.keyring 2024-03-11T20:52:24.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4075675345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T20:52:24.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: Updating smithi157:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:52:24.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:23 smithi157 ceph-mon[33359]: Updating smithi129:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T20:52:24.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.262+0000 7f1706c86700 1 -- 172.21.15.129:0/1523986854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1700106060 msgr2=0x7f170010dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:24.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.262+0000 7f1706c86700 1 --2- 172.21.15.129:0/1523986854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1700106060 0x7f170010dc30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009fc0 tx=0x7f16f800b330 comp rx=0 tx=0).stop 2024-03-11T20:52:24.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f1706c86700 1 -- 172.21.15.129:0/1523986854 shutdown_connections 2024-03-11T20:52:24.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f1706c86700 1 --2- 172.21.15.129:0/1523986854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1700106060 0x7f170010dc30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:24.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f1706c86700 1 --2- 172.21.15.129:0/1523986854 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f1700101740 0x7f1700101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:24.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f1706c86700 1 --2- 172.21.15.129:0/1523986854 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17001020f0 0x7f1700105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:24.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f1706c86700 1 -- 172.21.15.129:0/1523986854 >> 172.21.15.129:0/1523986854 conn(0x7f17000772e0 msgr2=0x7f17000fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:24.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f1706c86700 1 -- 172.21.15.129:0/1523986854 shutdown_connections 2024-03-11T20:52:24.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f1706c86700 1 -- 172.21.15.129:0/1523986854 wait complete. 2024-03-11T20:52:24.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f1706c86700 1 Processor -- start 2024-03-11T20:52:24.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f1706c86700 1 -- start start 2024-03-11T20:52:24.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f1706c86700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1700101740 0x7f17000fc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:24.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f1706c86700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17001020f0 0x7f17000fcd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:24.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f1706c86700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1700106060 0x7f17000fd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:24.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f1706c86700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1700110570 con 0x7f1700106060 2024-03-11T20:52:24.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f1706c86700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17001103f0 con 0x7f1700101740 2024-03-11T20:52:24.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f1706c86700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17001106f0 con 0x7f17001020f0 2024-03-11T20:52:24.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f16fffff700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17001020f0 0x7f17000fcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:24.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.263+0000 7f1705223700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1700106060 0x7f17000fd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:24.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.264+0000 7f16fffff700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17001020f0 0x7f17000fcd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:40322/0 (socket says 172.21.15.129:40322) 2024-03-11T20:52:24.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.264+0000 7f16fffff700 1 -- 172.21.15.129:0/2800389827 learned_addr learned my addr 172.21.15.129:0/2800389827 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:52:24.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.264+0000 7f1704a22700 1 --2- 172.21.15.129:0/2800389827 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1700101740 0x7f17000fc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:24.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.264+0000 7f1705223700 1 -- 172.21.15.129:0/2800389827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17001020f0 msgr2=0x7f17000fcd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:24.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.264+0000 7f1705223700 1 --2- 172.21.15.129:0/2800389827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17001020f0 0x7f17000fcd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:24.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.264+0000 7f1705223700 1 -- 172.21.15.129:0/2800389827 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1700101740 msgr2=0x7f17000fc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:24.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.264+0000 7f1705223700 1 --2- 172.21.15.129:0/2800389827 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1700101740 0x7f17000fc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:24.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.264+0000 7f1705223700 1 -- 172.21.15.129:0/2800389827 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17001a5ff0 con 0x7f1700106060 2024-03-11T20:52:24.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.264+0000 7f16fffff700 1 --2- 172.21.15.129:0/2800389827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17001020f0 0x7f17000fcd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:52:24.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.264+0000 7f1704a22700 1 --2- 172.21.15.129:0/2800389827 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1700101740 0x7f17000fc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:52:24.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.264+0000 7f1705223700 1 --2- 172.21.15.129:0/2800389827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1700106060 0x7f17000fd240 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f16f800b6c0 tx=0x7f16f800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:24.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.264+0000 7f16fdffb700 1 -- 172.21.15.129:0/2800389827 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8021070 con 0x7f1700106060 2024-03-11T20:52:24.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.264+0000 7f1706c86700 1 -- 172.21.15.129:0/2800389827 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1700197800 con 0x7f1700106060 2024-03-11T20:52:24.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.264+0000 7f16fdffb700 1 -- 172.21.15.129:0/2800389827 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f16f801d450 con 0x7f1700106060 2024-03-11T20:52:24.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.264+0000 7f16fdffb700 1 -- 172.21.15.129:0/2800389827 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f800cea0 con 0x7f1700106060 2024-03-11T20:52:24.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.264+0000 7f1706c86700 1 -- 172.21.15.129:0/2800389827 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1700197d90 con 0x7f1700106060 2024-03-11T20:52:24.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.265+0000 7f1706c86700 1 -- 172.21.15.129:0/2800389827 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17001024f0 con 0x7f1700106060 2024-03-11T20:52:24.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.266+0000 7f16fdffb700 1 -- 172.21.15.129:0/2800389827 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f16f8024070 con 0x7f1700106060 2024-03-11T20:52:24.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.266+0000 7f16fdffb700 1 --2- 172.21.15.129:0/2800389827 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f16f006de80 0x7f16f0070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:24.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.267+0000 7f16fdffb700 1 -- 172.21.15.129:0/2800389827 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f16f800f020 con 0x7f1700106060 2024-03-11T20:52:24.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.267+0000 7f1704a22700 1 --2- 172.21.15.129:0/2800389827 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f16f006de80 0x7f16f0070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:24.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.267+0000 7f1704a22700 1 --2- 172.21.15.129:0/2800389827 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f16f006de80 0x7f16f0070340 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f16f4005fd0 tx=0x7f16f4016040 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:24.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.276+0000 7f16fdffb700 1 -- 172.21.15.129:0/2800389827 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f16f802b020 con 0x7f1700106060 2024-03-11T20:52:24.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.477+0000 7f1706c86700 1 -- 172.21.15.129:0/2800389827 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1700197f60 con 0x7f1700106060 2024-03-11T20:52:24.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.477+0000 7f16fdffb700 1 -- 172.21.15.129:0/2800389827 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f16f8024360 con 0x7f1700106060 2024-03-11T20:52:24.473 INFO:teuthology.orchestra.run.smithi129.stdout:107374182438 2024-03-11T20:52:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.480+0000 7f1706c86700 1 -- 172.21.15.129:0/2800389827 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f16f006de80 msgr2=0x7f16f0070340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.480+0000 7f1706c86700 1 --2- 172.21.15.129:0/2800389827 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f16f006de80 0x7f16f0070340 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f16f4005fd0 tx=0x7f16f4016040 comp rx=0 tx=0).stop 2024-03-11T20:52:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.480+0000 7f1706c86700 1 -- 172.21.15.129:0/2800389827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1700106060 msgr2=0x7f17000fd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.480+0000 7f1706c86700 1 --2- 172.21.15.129:0/2800389827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1700106060 0x7f17000fd240 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f16f800b6c0 tx=0x7f16f800cc30 comp rx=0 tx=0).stop 2024-03-11T20:52:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.480+0000 7f1706c86700 1 -- 172.21.15.129:0/2800389827 shutdown_connections 2024-03-11T20:52:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.480+0000 7f1706c86700 1 --2- 172.21.15.129:0/2800389827 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f16f006de80 0x7f16f0070340 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.480+0000 7f1706c86700 1 --2- 172.21.15.129:0/2800389827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1700106060 0x7f17000fd240 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.480+0000 7f1706c86700 1 --2- 172.21.15.129:0/2800389827 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1700101740 0x7f17000fc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.480+0000 7f1706c86700 1 --2- 172.21.15.129:0/2800389827 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f17001020f0 0x7f17000fcd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.480+0000 7f1706c86700 1 -- 172.21.15.129:0/2800389827 >> 172.21.15.129:0/2800389827 conn(0x7f17000772e0 msgr2=0x7f17001005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.480+0000 7f1706c86700 1 -- 172.21.15.129:0/2800389827 shutdown_connections 2024-03-11T20:52:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:24.480+0000 7f1706c86700 1 -- 172.21.15.129:0/2800389827 wait complete. 2024-03-11T20:52:24.950 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574896 for osd.1 2024-03-11T20:52:24.950 DEBUG:teuthology.parallel:result is None 2024-03-11T20:52:25.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:24 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:25.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:24 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:25.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:24 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:25.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:24 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:25.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:24 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:25.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:24 smithi129 ceph-mon[30184]: Deploying daemon alertmanager.a on smithi129 2024-03-11T20:52:25.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2800389827' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T20:52:25.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:24 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:25.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:24 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:25.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:24 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:25.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:24 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:25.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:24 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:25.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:24 smithi129 ceph-mon[42066]: Deploying daemon alertmanager.a on smithi129 2024-03-11T20:52:25.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2800389827' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T20:52:25.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:24 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:24 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:24 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:24 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:24 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:24 smithi157 ceph-mon[33359]: Deploying daemon alertmanager.a on smithi129 2024-03-11T20:52:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2800389827' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T20:52:25.723 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182434 got 107374182438 for osd.3 2024-03-11T20:52:25.723 DEBUG:teuthology.parallel:result is None 2024-03-11T20:52:25.724 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T20:52:25.724 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T20:52:25.976 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:25.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:25 smithi129 ceph-mon[30184]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:25.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:25 smithi129 ceph-mon[42066]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:25 smithi157 ceph-mon[33359]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:26.355 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:52:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:52:27.974 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:27 smithi129 ceph-mon[30184]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:27.975 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:27 smithi129 ceph-mon[42066]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:28.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.978+0000 7f76e25f4700 1 -- 172.21.15.129:0/2200486373 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76dc101680 msgr2=0x7f76dc10db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:28.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.978+0000 7f76e25f4700 1 --2- 172.21.15.129:0/2200486373 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76dc101680 0x7f76dc10db60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f76d4009f90 tx=0x7f76d400b330 comp rx=0 tx=0).stop 2024-03-11T20:52:28.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.978+0000 7f76e25f4700 1 -- 172.21.15.129:0/2200486373 shutdown_connections 2024-03-11T20:52:28.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.978+0000 7f76e25f4700 1 --2- 172.21.15.129:0/2200486373 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76dc101680 0x7f76dc10db60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:28.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.978+0000 7f76e25f4700 1 --2- 172.21.15.129:0/2200486373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76dc106d20 0x7f76dc107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:28.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.978+0000 7f76e25f4700 1 --2- 172.21.15.129:0/2200486373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc100ce0 0x7f76dc101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:28.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.978+0000 7f76e25f4700 1 -- 172.21.15.129:0/2200486373 >> 172.21.15.129:0/2200486373 conn(0x7f76dc077c10 msgr2=0x7f76dc078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:28.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.978+0000 7f76e25f4700 1 -- 172.21.15.129:0/2200486373 shutdown_connections 2024-03-11T20:52:28.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.978+0000 7f76e25f4700 1 -- 172.21.15.129:0/2200486373 wait complete. 2024-03-11T20:52:28.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.979+0000 7f76e25f4700 1 Processor -- start 2024-03-11T20:52:28.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.979+0000 7f76e25f4700 1 -- start start 2024-03-11T20:52:28.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.979+0000 7f76e25f4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76dc100ce0 0x7f76dc19c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:28.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.979+0000 7f76e25f4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76dc101680 0x7f76dc19cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:28.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.979+0000 7f76e25f4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc106d20 0x7f76dc196740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:28.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.979+0000 7f76e25f4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc110500 con 0x7f76dc106d20 2024-03-11T20:52:28.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.979+0000 7f76e25f4700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc110380 con 0x7f76dc100ce0 2024-03-11T20:52:28.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.979+0000 7f76e25f4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc110680 con 0x7f76dc101680 2024-03-11T20:52:28.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76e0b91700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc106d20 0x7f76dc196740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:28.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76e0b91700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc106d20 0x7f76dc196740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36172/0 (socket says 172.21.15.129:36172) 2024-03-11T20:52:28.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76e0b91700 1 -- 172.21.15.129:0/1805603564 learned_addr learned my addr 172.21.15.129:0/1805603564 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:52:28.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76db7fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76dc101680 0x7f76dc19cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:28.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76e0b91700 1 -- 172.21.15.129:0/1805603564 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76dc101680 msgr2=0x7f76dc19cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:28.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76e0b91700 1 --2- 172.21.15.129:0/1805603564 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76dc101680 0x7f76dc19cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:28.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76e0b91700 1 -- 172.21.15.129:0/1805603564 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76dc100ce0 msgr2=0x7f76dc19c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:28.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76dbfff700 1 --2- 172.21.15.129:0/1805603564 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76dc100ce0 0x7f76dc19c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:28.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76e0b91700 1 --2- 172.21.15.129:0/1805603564 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76dc100ce0 0x7f76dc19c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:28.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76e0b91700 1 -- 172.21.15.129:0/1805603564 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76dc197020 con 0x7f76dc106d20 2024-03-11T20:52:28.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76dbfff700 1 --2- 172.21.15.129:0/1805603564 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76dc100ce0 0x7f76dc19c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T20:52:28.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76e0b91700 1 --2- 172.21.15.129:0/1805603564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc106d20 0x7f76dc196740 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f76d400cd00 tx=0x7f76d400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:28.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76d97fa700 1 -- 172.21.15.129:0/1805603564 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d4020070 con 0x7f76dc106d20 2024-03-11T20:52:28.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76d97fa700 1 -- 172.21.15.129:0/1805603564 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f76d401a4c0 con 0x7f76dc106d20 2024-03-11T20:52:28.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76e25f4700 1 -- 172.21.15.129:0/1805603564 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76dc1972b0 con 0x7f76dc106d20 2024-03-11T20:52:28.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76d97fa700 1 -- 172.21.15.129:0/1805603564 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d40052a0 con 0x7f76dc106d20 2024-03-11T20:52:28.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.980+0000 7f76e25f4700 1 -- 172.21.15.129:0/1805603564 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76dc069a90 con 0x7f76dc106d20 2024-03-11T20:52:28.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.981+0000 7f76e25f4700 1 -- 172.21.15.129:0/1805603564 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76dc1023d0 con 0x7f76dc106d20 2024-03-11T20:52:28.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.982+0000 7f76d97fa700 1 -- 172.21.15.129:0/1805603564 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f76d4007ac0 con 0x7f76dc106d20 2024-03-11T20:52:28.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.982+0000 7f76d97fa700 1 --2- 172.21.15.129:0/1805603564 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f76c406de80 0x7f76c4070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:28.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.982+0000 7f76d97fa700 1 -- 172.21.15.129:0/1805603564 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f76d4090bb0 con 0x7f76dc106d20 2024-03-11T20:52:28.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.982+0000 7f76dbfff700 1 --2- 172.21.15.129:0/1805603564 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f76c406de80 0x7f76c4070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:28.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.983+0000 7f76dbfff700 1 --2- 172.21.15.129:0/1805603564 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f76c406de80 0x7f76c4070340 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f76cc00baa0 tx=0x7f76cc005db0 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:28.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:27.987+0000 7f76d97fa700 1 -- 172.21.15.129:0/1805603564 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f76d405e090 con 0x7f76dc106d20 2024-03-11T20:52:28.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:28.189+0000 7f76e25f4700 1 -- 172.21.15.129:0/1805603564 --> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f76dc069cf0 con 0x7f76c406de80 2024-03-11T20:52:28.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:28.199+0000 7f76d97fa700 1 -- 172.21.15.129:0/1805603564 <== mgr.24497 v2:172.21.15.129:6800/1191021258 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346472 (secure 0 0 0) 0x7f76dc069cf0 con 0x7f76c406de80 2024-03-11T20:52:28.199 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:52:28.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:28.202+0000 7f76e25f4700 1 -- 172.21.15.129:0/1805603564 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f76c406de80 msgr2=0x7f76c4070340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:28.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:28.202+0000 7f76e25f4700 1 --2- 172.21.15.129:0/1805603564 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f76c406de80 0x7f76c4070340 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f76cc00baa0 tx=0x7f76cc005db0 comp rx=0 tx=0).stop 2024-03-11T20:52:28.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:28.202+0000 7f76e25f4700 1 -- 172.21.15.129:0/1805603564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc106d20 msgr2=0x7f76dc196740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:28.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:28.202+0000 7f76e25f4700 1 --2- 172.21.15.129:0/1805603564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc106d20 0x7f76dc196740 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f76d400cd00 tx=0x7f76d400cd30 comp rx=0 tx=0).stop 2024-03-11T20:52:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:28.203+0000 7f76e25f4700 1 -- 172.21.15.129:0/1805603564 shutdown_connections 2024-03-11T20:52:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:28.203+0000 7f76e25f4700 1 --2- 172.21.15.129:0/1805603564 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f76c406de80 0x7f76c4070340 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:28.203+0000 7f76e25f4700 1 --2- 172.21.15.129:0/1805603564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76dc106d20 0x7f76dc196740 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:28.203+0000 7f76e25f4700 1 --2- 172.21.15.129:0/1805603564 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76dc100ce0 0x7f76dc19c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:28.203+0000 7f76e25f4700 1 --2- 172.21.15.129:0/1805603564 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f76dc101680 0x7f76dc19cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:28.203+0000 7f76e25f4700 1 -- 172.21.15.129:0/1805603564 >> 172.21.15.129:0/1805603564 conn(0x7f76dc077c10 msgr2=0x7f76dc0ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:28.203+0000 7f76e25f4700 1 -- 172.21.15.129:0/1805603564 shutdown_connections 2024-03-11T20:52:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:28.203+0000 7f76e25f4700 1 -- 172.21.15.129:0/1805603564 wait complete. 2024-03-11T20:52:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-03-11T20:52:28.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:27 smithi157 ceph-mon[33359]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:29.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:28 smithi129 ceph-mon[30184]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:52:29.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:28 smithi129 ceph-mon[42066]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:52:29.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:28 smithi157 ceph-mon[33359]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:52:29.481 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-03-11T20:52:26.724744+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":998,"num_read_kb":844,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":600,"ondisk_log_size":600,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236724,"kb_used_data":8092,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724492,"statfs":{"total":767960285184,"available":767717879808,"internally_reserved":0,"allocated":8286208,"data_stored":4481216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":234016306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002637"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707352+0000","last_change":"2024-03-11T20:51:17.761571+0000","last_active":"2024-03-11T20:52:00.707352+0000","last_peered":"2024-03-11T20:52:00.707352+0000","last_clean":"2024-03-11T20:52:00.707352+0000","last_became_active":"2024-03-11T20:51:17.761282+0000","last_became_peered":"2024-03-11T20:51:17.761282+0000","last_unstale":"2024-03-11T20:52:00.707352+0000","last_undegraded":"2024-03-11T20:52:00.707352+0000","last_fullsized":"2024-03-11T20:52:00.707352+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:17:34.237398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344609+0000","last_change":"2024-03-11T20:51:15.748116+0000","last_active":"2024-03-11T20:52:01.344609+0000","last_peered":"2024-03-11T20:52:01.344609+0000","last_clean":"2024-03-11T20:52:01.344609+0000","last_became_active":"2024-03-11T20:51:15.747469+0000","last_became_peered":"2024-03-11T20:51:15.747469+0000","last_unstale":"2024-03-11T20:52:01.344609+0000","last_undegraded":"2024-03-11T20:52:01.344609+0000","last_fullsized":"2024-03-11T20:52:01.344609+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:19:54.509317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345051+0000","last_change":"2024-03-11T20:51:11.724723+0000","last_active":"2024-03-11T20:52:01.345051+0000","last_peered":"2024-03-11T20:52:01.345051+0000","last_clean":"2024-03-11T20:52:01.345051+0000","last_became_active":"2024-03-11T20:51:11.723846+0000","last_became_peered":"2024-03-11T20:51:11.723846+0000","last_unstale":"2024-03-11T20:52:01.345051+0000","last_undegraded":"2024-03-11T20:52:01.345051+0000","last_fullsized":"2024-03-11T20:52:01.345051+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:29:31.025832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707277+0000","last_change":"2024-03-11T20:51:13.735985+0000","last_active":"2024-03-11T20:52:00.707277+0000","last_peered":"2024-03-11T20:52:00.707277+0000","last_clean":"2024-03-11T20:52:00.707277+0000","last_became_active":"2024-03-11T20:51:13.735016+0000","last_became_peered":"2024-03-11T20:51:13.735016+0000","last_unstale":"2024-03-11T20:52:00.707277+0000","last_undegraded":"2024-03-11T20:52:00.707277+0000","last_fullsized":"2024-03-11T20:52:00.707277+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:54:17.557137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710395+0000","last_change":"2024-03-11T20:51:17.891186+0000","last_active":"2024-03-11T20:52:00.710395+0000","last_peered":"2024-03-11T20:52:00.710395+0000","last_clean":"2024-03-11T20:52:00.710395+0000","last_became_active":"2024-03-11T20:51:17.890804+0000","last_became_peered":"2024-03-11T20:51:17.890804+0000","last_unstale":"2024-03-11T20:52:00.710395+0000","last_undegraded":"2024-03-11T20:52:00.710395+0000","last_fullsized":"2024-03-11T20:52:00.710395+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:52:39.048603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707279+0000","last_change":"2024-03-11T20:51:15.745388+0000","last_active":"2024-03-11T20:52:00.707279+0000","last_peered":"2024-03-11T20:52:00.707279+0000","last_clean":"2024-03-11T20:52:00.707279+0000","last_became_active":"2024-03-11T20:51:15.745100+0000","last_became_peered":"2024-03-11T20:51:15.745100+0000","last_unstale":"2024-03-11T20:52:00.707279+0000","last_undegraded":"2024-03-11T20:52:00.707279+0000","last_fullsized":"2024-03-11T20:52:00.707279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:14:47.981855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.711724+0000","last_change":"2024-03-11T20:51:11.721114+0000","last_active":"2024-03-11T20:52:00.711724+0000","last_peered":"2024-03-11T20:52:00.711724+0000","last_clean":"2024-03-11T20:52:00.711724+0000","last_became_active":"2024-03-11T20:51:11.720862+0000","last_became_peered":"2024-03-11T20:51:11.720862+0000","last_unstale":"2024-03-11T20:52:00.711724+0000","last_undegraded":"2024-03-11T20:52:00.711724+0000","last_fullsized":"2024-03-11T20:52:00.711724+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:27:34.180819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.712570+0000","last_change":"2024-03-11T20:51:13.732877+0000","last_active":"2024-03-11T20:52:00.712570+0000","last_peered":"2024-03-11T20:52:00.712570+0000","last_clean":"2024-03-11T20:52:00.712570+0000","last_became_active":"2024-03-11T20:51:13.732570+0000","last_became_peered":"2024-03-11T20:51:13.732570+0000","last_unstale":"2024-03-11T20:52:00.712570+0000","last_undegraded":"2024-03-11T20:52:00.712570+0000","last_fullsized":"2024-03-11T20:52:00.712570+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:35:01.028096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343978+0000","last_change":"2024-03-11T20:51:17.757301+0000","last_active":"2024-03-11T20:52:01.343978+0000","last_peered":"2024-03-11T20:52:01.343978+0000","last_clean":"2024-03-11T20:52:01.343978+0000","last_became_active":"2024-03-11T20:51:17.757007+0000","last_became_peered":"2024-03-11T20:51:17.757007+0000","last_unstale":"2024-03-11T20:52:01.343978+0000","last_undegraded":"2024-03-11T20:52:01.343978+0000","last_fullsized":"2024-03-11T20:52:01.343978+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:39:43.437020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821671+0000","last_change":"2024-03-11T20:51:15.750410+0000","last_active":"2024-03-11T20:52:00.821671+0000","last_peered":"2024-03-11T20:52:00.821671+0000","last_clean":"2024-03-11T20:52:00.821671+0000","last_became_active":"2024-03-11T20:51:15.749748+0000","last_became_peered":"2024-03-11T20:51:15.749748+0000","last_unstale":"2024-03-11T20:52:00.821671+0000","last_undegraded":"2024-03-11T20:52:00.821671+0000","last_fullsized":"2024-03-11T20:52:00.821671+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:35:15.056079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821911+0000","last_change":"2024-03-11T20:51:11.730768+0000","last_active":"2024-03-11T20:52:00.821911+0000","last_peered":"2024-03-11T20:52:00.821911+0000","last_clean":"2024-03-11T20:52:00.821911+0000","last_became_active":"2024-03-11T20:51:11.730460+0000","last_became_peered":"2024-03-11T20:51:11.730460+0000","last_unstale":"2024-03-11T20:52:00.821911+0000","last_undegraded":"2024-03-11T20:52:00.821911+0000","last_fullsized":"2024-03-11T20:52:00.821911+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:26:03.386745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707520+0000","last_change":"2024-03-11T20:51:13.731900+0000","last_active":"2024-03-11T20:52:00.707520+0000","last_peered":"2024-03-11T20:52:00.707520+0000","last_clean":"2024-03-11T20:52:00.707520+0000","last_became_active":"2024-03-11T20:51:13.731261+0000","last_became_peered":"2024-03-11T20:51:13.731261+0000","last_unstale":"2024-03-11T20:52:00.707520+0000","last_undegraded":"2024-03-11T20:52:00.707520+0000","last_fullsized":"2024-03-11T20:52:00.707520+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:05:37.179737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821436+0000","last_change":"2024-03-11T20:51:17.754951+0000","last_active":"2024-03-11T20:52:00.821436+0000","last_peered":"2024-03-11T20:52:00.821436+0000","last_clean":"2024-03-11T20:52:00.821436+0000","last_became_active":"2024-03-11T20:51:17.754618+0000","last_became_peered":"2024-03-11T20:51:17.754618+0000","last_unstale":"2024-03-11T20:52:00.821436+0000","last_undegraded":"2024-03-11T20:52:00.821436+0000","last_fullsized":"2024-03-11T20:52:00.821436+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:06:16.061476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828511+0000","last_change":"2024-03-11T20:51:15.744164+0000","last_active":"2024-03-11T20:52:00.828511+0000","last_peered":"2024-03-11T20:52:00.828511+0000","last_clean":"2024-03-11T20:52:00.828511+0000","last_became_active":"2024-03-11T20:51:15.743883+0000","last_became_peered":"2024-03-11T20:51:15.743883+0000","last_unstale":"2024-03-11T20:52:00.828511+0000","last_undegraded":"2024-03-11T20:52:00.828511+0000","last_fullsized":"2024-03-11T20:52:00.828511+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:26:17.521657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821819+0000","last_change":"2024-03-11T20:51:11.731273+0000","last_active":"2024-03-11T20:52:00.821819+0000","last_peered":"2024-03-11T20:52:00.821819+0000","last_clean":"2024-03-11T20:52:00.821819+0000","last_became_active":"2024-03-11T20:51:11.730970+0000","last_became_peered":"2024-03-11T20:51:11.730970+0000","last_unstale":"2024-03-11T20:52:00.821819+0000","last_undegraded":"2024-03-11T20:52:00.821819+0000","last_fullsized":"2024-03-11T20:52:00.821819+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:52:55.019056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.708794+0000","last_change":"2024-03-11T20:51:13.735898+0000","last_active":"2024-03-11T20:52:00.708794+0000","last_peered":"2024-03-11T20:52:00.708794+0000","last_clean":"2024-03-11T20:52:00.708794+0000","last_became_active":"2024-03-11T20:51:13.735549+0000","last_became_peered":"2024-03-11T20:51:13.735549+0000","last_unstale":"2024-03-11T20:52:00.708794+0000","last_undegraded":"2024-03-11T20:52:00.708794+0000","last_fullsized":"2024-03-11T20:52:00.708794+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:12:07.636129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709364+0000","last_change":"2024-03-11T20:51:17.888991+0000","last_active":"2024-03-11T20:52:00.709364+0000","last_peered":"2024-03-11T20:52:00.709364+0000","last_clean":"2024-03-11T20:52:00.709364+0000","last_became_active":"2024-03-11T20:51:17.888691+0000","last_became_peered":"2024-03-11T20:51:17.888691+0000","last_unstale":"2024-03-11T20:52:00.709364+0000","last_undegraded":"2024-03-11T20:52:00.709364+0000","last_fullsized":"2024-03-11T20:52:00.709364+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:18:23.025134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345698+0000","last_change":"2024-03-11T20:51:15.744293+0000","last_active":"2024-03-11T20:52:01.345698+0000","last_peered":"2024-03-11T20:52:01.345698+0000","last_clean":"2024-03-11T20:52:01.345698+0000","last_became_active":"2024-03-11T20:51:15.743950+0000","last_became_peered":"2024-03-11T20:51:15.743950+0000","last_unstale":"2024-03-11T20:52:01.345698+0000","last_undegraded":"2024-03-11T20:52:01.345698+0000","last_fullsized":"2024-03-11T20:52:01.345698+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:28:05.486252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344833+0000","last_change":"2024-03-11T20:51:11.730277+0000","last_active":"2024-03-11T20:52:01.344833+0000","last_peered":"2024-03-11T20:52:01.344833+0000","last_clean":"2024-03-11T20:52:01.344833+0000","last_became_active":"2024-03-11T20:51:11.729868+0000","last_became_peered":"2024-03-11T20:51:11.729868+0000","last_unstale":"2024-03-11T20:52:01.344833+0000","last_undegraded":"2024-03-11T20:52:01.344833+0000","last_fullsized":"2024-03-11T20:52:01.344833+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:28:00.331652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707532+0000","last_change":"2024-03-11T20:51:13.736663+0000","last_active":"2024-03-11T20:52:00.707532+0000","last_peered":"2024-03-11T20:52:00.707532+0000","last_clean":"2024-03-11T20:52:00.707532+0000","last_became_active":"2024-03-11T20:51:13.736316+0000","last_became_peered":"2024-03-11T20:51:13.736316+0000","last_unstale":"2024-03-11T20:52:00.707532+0000","last_undegraded":"2024-03-11T20:52:00.707532+0000","last_fullsized":"2024-03-11T20:52:00.707532+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:03:40.249599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.712433+0000","last_change":"2024-03-11T20:51:17.888988+0000","last_active":"2024-03-11T20:52:00.712433+0000","last_peered":"2024-03-11T20:52:00.712433+0000","last_clean":"2024-03-11T20:52:00.712433+0000","last_became_active":"2024-03-11T20:51:17.888587+0000","last_became_peered":"2024-03-11T20:51:17.888587+0000","last_unstale":"2024-03-11T20:52:00.712433+0000","last_undegraded":"2024-03-11T20:52:00.712433+0000","last_fullsized":"2024-03-11T20:52:00.712433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:49:49.768783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709191+0000","last_change":"2024-03-11T20:51:15.747529+0000","last_active":"2024-03-11T20:52:00.709191+0000","last_peered":"2024-03-11T20:52:00.709191+0000","last_clean":"2024-03-11T20:52:00.709191+0000","last_became_active":"2024-03-11T20:51:15.747259+0000","last_became_peered":"2024-03-11T20:51:15.747259+0000","last_unstale":"2024-03-11T20:52:00.709191+0000","last_undegraded":"2024-03-11T20:52:00.709191+0000","last_fullsized":"2024-03-11T20:52:00.709191+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:21:01.320072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707586+0000","last_change":"2024-03-11T20:51:11.727831+0000","last_active":"2024-03-11T20:52:00.707586+0000","last_peered":"2024-03-11T20:52:00.707586+0000","last_clean":"2024-03-11T20:52:00.707586+0000","last_became_active":"2024-03-11T20:51:11.727399+0000","last_became_peered":"2024-03-11T20:51:11.727399+0000","last_unstale":"2024-03-11T20:52:00.707586+0000","last_undegraded":"2024-03-11T20:52:00.707586+0000","last_fullsized":"2024-03-11T20:52:00.707586+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:54:05.940769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":52,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710984+0000","last_change":"2024-03-11T20:51:13.737828+0000","last_active":"2024-03-11T20:52:00.710984+0000","last_peered":"2024-03-11T20:52:00.710984+0000","last_clean":"2024-03-11T20:52:00.710984+0000","last_became_active":"2024-03-11T20:51:13.737487+0000","last_became_peered":"2024-03-11T20:51:13.737487+0000","last_unstale":"2024-03-11T20:52:00.710984+0000","last_undegraded":"2024-03-11T20:52:00.710984+0000","last_fullsized":"2024-03-11T20:52:00.710984+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:36:31.772982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828670+0000","last_change":"2024-03-11T20:51:17.756348+0000","last_active":"2024-03-11T20:52:00.828670+0000","last_peered":"2024-03-11T20:52:00.828670+0000","last_clean":"2024-03-11T20:52:00.828670+0000","last_became_active":"2024-03-11T20:51:17.755977+0000","last_became_peered":"2024-03-11T20:51:17.755977+0000","last_unstale":"2024-03-11T20:52:00.828670+0000","last_undegraded":"2024-03-11T20:52:00.828670+0000","last_fullsized":"2024-03-11T20:52:00.828670+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:57:52.912998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:11.787432+0000","last_change":"2024-03-11T20:51:15.749045+0000","last_active":"2024-03-11T20:52:11.787432+0000","last_peered":"2024-03-11T20:52:11.787432+0000","last_clean":"2024-03-11T20:52:11.787432+0000","last_became_active":"2024-03-11T20:51:15.748731+0000","last_became_peered":"2024-03-11T20:51:15.748731+0000","last_unstale":"2024-03-11T20:52:11.787432+0000","last_undegraded":"2024-03-11T20:52:11.787432+0000","last_fullsized":"2024-03-11T20:52:11.787432+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:32:17.002950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344675+0000","last_change":"2024-03-11T20:51:11.729154+0000","last_active":"2024-03-11T20:52:01.344675+0000","last_peered":"2024-03-11T20:52:01.344675+0000","last_clean":"2024-03-11T20:52:01.344675+0000","last_became_active":"2024-03-11T20:51:11.728835+0000","last_became_peered":"2024-03-11T20:51:11.728835+0000","last_unstale":"2024-03-11T20:52:01.344675+0000","last_undegraded":"2024-03-11T20:52:01.344675+0000","last_fullsized":"2024-03-11T20:52:01.344675+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:52:02.829643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345323+0000","last_change":"2024-03-11T20:51:14.071941+0000","last_active":"2024-03-11T20:52:01.345323+0000","last_peered":"2024-03-11T20:52:01.345323+0000","last_clean":"2024-03-11T20:52:01.345323+0000","last_became_active":"2024-03-11T20:51:14.071283+0000","last_became_peered":"2024-03-11T20:51:14.071283+0000","last_unstale":"2024-03-11T20:52:01.345323+0000","last_undegraded":"2024-03-11T20:52:01.345323+0000","last_fullsized":"2024-03-11T20:52:01.345323+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:29:28.919667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345852+0000","last_change":"2024-03-11T20:51:17.754330+0000","last_active":"2024-03-11T20:52:01.345852+0000","last_peered":"2024-03-11T20:52:01.345852+0000","last_clean":"2024-03-11T20:52:01.345852+0000","last_became_active":"2024-03-11T20:51:17.753989+0000","last_became_peered":"2024-03-11T20:51:17.753989+0000","last_unstale":"2024-03-11T20:52:01.345852+0000","last_undegraded":"2024-03-11T20:52:01.345852+0000","last_fullsized":"2024-03-11T20:52:01.345852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:28:56.527627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345512+0000","last_change":"2024-03-11T20:51:15.748177+0000","last_active":"2024-03-11T20:52:01.345512+0000","last_peered":"2024-03-11T20:52:01.345512+0000","last_clean":"2024-03-11T20:52:01.345512+0000","last_became_active":"2024-03-11T20:51:15.747874+0000","last_became_peered":"2024-03-11T20:51:15.747874+0000","last_unstale":"2024-03-11T20:52:01.345512+0000","last_undegraded":"2024-03-11T20:52:01.345512+0000","last_fullsized":"2024-03-11T20:52:01.345512+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:07:50.572831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709418+0000","last_change":"2024-03-11T20:51:11.724549+0000","last_active":"2024-03-11T20:52:00.709418+0000","last_peered":"2024-03-11T20:52:00.709418+0000","last_clean":"2024-03-11T20:52:00.709418+0000","last_became_active":"2024-03-11T20:51:11.724267+0000","last_became_peered":"2024-03-11T20:51:11.724267+0000","last_unstale":"2024-03-11T20:52:00.709418+0000","last_undegraded":"2024-03-11T20:52:00.709418+0000","last_fullsized":"2024-03-11T20:52:00.709418+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:15:01.634207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":66,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344346+0000","last_change":"2024-03-11T20:51:31.769074+0000","last_active":"2024-03-11T20:52:01.344346+0000","last_peered":"2024-03-11T20:52:01.344346+0000","last_clean":"2024-03-11T20:52:01.344346+0000","last_became_active":"2024-03-11T20:51:31.757275+0000","last_became_peered":"2024-03-11T20:51:31.757275+0000","last_unstale":"2024-03-11T20:52:01.344346+0000","last_undegraded":"2024-03-11T20:52:01.344346+0000","last_fullsized":"2024-03-11T20:52:01.344346+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:53:29.566961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828500+0000","last_change":"2024-03-11T20:51:17.756239+0000","last_active":"2024-03-11T20:52:00.828500+0000","last_peered":"2024-03-11T20:52:00.828500+0000","last_clean":"2024-03-11T20:52:00.828500+0000","last_became_active":"2024-03-11T20:51:17.755925+0000","last_became_peered":"2024-03-11T20:51:17.755925+0000","last_unstale":"2024-03-11T20:52:00.828500+0000","last_undegraded":"2024-03-11T20:52:00.828500+0000","last_fullsized":"2024-03-11T20:52:00.828500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:35.600247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343691+0000","last_change":"2024-03-11T20:51:15.745881+0000","last_active":"2024-03-11T20:52:01.343691+0000","last_peered":"2024-03-11T20:52:01.343691+0000","last_clean":"2024-03-11T20:52:01.343691+0000","last_became_active":"2024-03-11T20:51:15.745382+0000","last_became_peered":"2024-03-11T20:51:15.745382+0000","last_unstale":"2024-03-11T20:52:01.343691+0000","last_undegraded":"2024-03-11T20:52:01.343691+0000","last_fullsized":"2024-03-11T20:52:01.343691+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:10:17.366734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344971+0000","last_change":"2024-03-11T20:51:11.724862+0000","last_active":"2024-03-11T20:52:01.344971+0000","last_peered":"2024-03-11T20:52:01.344971+0000","last_clean":"2024-03-11T20:52:01.344971+0000","last_became_active":"2024-03-11T20:51:11.724584+0000","last_became_peered":"2024-03-11T20:51:11.724584+0000","last_unstale":"2024-03-11T20:52:01.344971+0000","last_undegraded":"2024-03-11T20:52:01.344971+0000","last_fullsized":"2024-03-11T20:52:01.344971+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:38:02.538772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":35,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709522+0000","last_change":"2024-03-11T20:51:13.737953+0000","last_active":"2024-03-11T20:52:00.709522+0000","last_peered":"2024-03-11T20:52:00.709522+0000","last_clean":"2024-03-11T20:52:00.709522+0000","last_became_active":"2024-03-11T20:51:13.737476+0000","last_became_peered":"2024-03-11T20:51:13.737476+0000","last_unstale":"2024-03-11T20:52:00.709522+0000","last_undegraded":"2024-03-11T20:52:00.709522+0000","last_fullsized":"2024-03-11T20:52:00.709522+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:53:15.620913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710196+0000","last_change":"2024-03-11T20:51:17.889382+0000","last_active":"2024-03-11T20:52:00.710196+0000","last_peered":"2024-03-11T20:52:00.710196+0000","last_clean":"2024-03-11T20:52:00.710196+0000","last_became_active":"2024-03-11T20:51:17.889067+0000","last_became_peered":"2024-03-11T20:51:17.889067+0000","last_unstale":"2024-03-11T20:52:00.710196+0000","last_undegraded":"2024-03-11T20:52:00.710196+0000","last_fullsized":"2024-03-11T20:52:00.710196+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:53:45.136250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828691+0000","last_change":"2024-03-11T20:51:15.746251+0000","last_active":"2024-03-11T20:52:00.828691+0000","last_peered":"2024-03-11T20:52:00.828691+0000","last_clean":"2024-03-11T20:52:00.828691+0000","last_became_active":"2024-03-11T20:51:15.745943+0000","last_became_peered":"2024-03-11T20:51:15.745943+0000","last_unstale":"2024-03-11T20:52:00.828691+0000","last_undegraded":"2024-03-11T20:52:00.828691+0000","last_fullsized":"2024-03-11T20:52:00.828691+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:40.441153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821990+0000","last_change":"2024-03-11T20:51:11.724708+0000","last_active":"2024-03-11T20:52:00.821990+0000","last_peered":"2024-03-11T20:52:00.821990+0000","last_clean":"2024-03-11T20:52:00.821990+0000","last_became_active":"2024-03-11T20:51:11.724386+0000","last_became_peered":"2024-03-11T20:51:11.724386+0000","last_unstale":"2024-03-11T20:52:00.821990+0000","last_undegraded":"2024-03-11T20:52:00.821990+0000","last_fullsized":"2024-03-11T20:52:00.821990+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:43.057620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.712446+0000","last_change":"2024-03-11T20:51:14.072821+0000","last_active":"2024-03-11T20:52:00.712446+0000","last_peered":"2024-03-11T20:52:00.712446+0000","last_clean":"2024-03-11T20:52:00.712446+0000","last_became_active":"2024-03-11T20:51:14.072513+0000","last_became_peered":"2024-03-11T20:51:14.072513+0000","last_unstale":"2024-03-11T20:52:00.712446+0000","last_undegraded":"2024-03-11T20:52:00.712446+0000","last_fullsized":"2024-03-11T20:52:00.712446+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:35:06.392468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343752+0000","last_change":"2024-03-11T20:51:17.763399+0000","last_active":"2024-03-11T20:52:01.343752+0000","last_peered":"2024-03-11T20:52:01.343752+0000","last_clean":"2024-03-11T20:52:01.343752+0000","last_became_active":"2024-03-11T20:51:17.763064+0000","last_became_peered":"2024-03-11T20:51:17.763064+0000","last_unstale":"2024-03-11T20:52:01.343752+0000","last_undegraded":"2024-03-11T20:52:01.343752+0000","last_fullsized":"2024-03-11T20:52:01.343752+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:27:50.694671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710490+0000","last_change":"2024-03-11T20:51:15.747901+0000","last_active":"2024-03-11T20:52:00.710490+0000","last_peered":"2024-03-11T20:52:00.710490+0000","last_clean":"2024-03-11T20:52:00.710490+0000","last_became_active":"2024-03-11T20:51:15.747588+0000","last_became_peered":"2024-03-11T20:51:15.747588+0000","last_unstale":"2024-03-11T20:52:00.710490+0000","last_undegraded":"2024-03-11T20:52:00.710490+0000","last_fullsized":"2024-03-11T20:52:00.710490+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:46:49.002126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828774+0000","last_change":"2024-03-11T20:51:11.729623+0000","last_active":"2024-03-11T20:52:00.828774+0000","last_peered":"2024-03-11T20:52:00.828774+0000","last_clean":"2024-03-11T20:52:00.828774+0000","last_became_active":"2024-03-11T20:51:11.729216+0000","last_became_peered":"2024-03-11T20:51:11.729216+0000","last_unstale":"2024-03-11T20:52:00.828774+0000","last_undegraded":"2024-03-11T20:52:00.828774+0000","last_fullsized":"2024-03-11T20:52:00.828774+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:39:32.506304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344995+0000","last_change":"2024-03-11T20:51:13.735847+0000","last_active":"2024-03-11T20:52:01.344995+0000","last_peered":"2024-03-11T20:52:01.344995+0000","last_clean":"2024-03-11T20:52:01.344995+0000","last_became_active":"2024-03-11T20:51:13.733822+0000","last_became_peered":"2024-03-11T20:51:13.733822+0000","last_unstale":"2024-03-11T20:52:01.344995+0000","last_undegraded":"2024-03-11T20:52:01.344995+0000","last_fullsized":"2024-03-11T20:52:01.344995+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:13:19.355129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710007+0000","last_change":"2024-03-11T20:51:17.763013+0000","last_active":"2024-03-11T20:52:00.710007+0000","last_peered":"2024-03-11T20:52:00.710007+0000","last_clean":"2024-03-11T20:52:00.710007+0000","last_became_active":"2024-03-11T20:51:17.761593+0000","last_became_peered":"2024-03-11T20:51:17.761593+0000","last_unstale":"2024-03-11T20:52:00.710007+0000","last_undegraded":"2024-03-11T20:52:00.710007+0000","last_fullsized":"2024-03-11T20:52:00.710007+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:51:12.808744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344963+0000","last_change":"2024-03-11T20:51:15.749619+0000","last_active":"2024-03-11T20:52:01.344963+0000","last_peered":"2024-03-11T20:52:01.344963+0000","last_clean":"2024-03-11T20:52:01.344963+0000","last_became_active":"2024-03-11T20:51:15.749319+0000","last_became_peered":"2024-03-11T20:51:15.749319+0000","last_unstale":"2024-03-11T20:52:01.344963+0000","last_undegraded":"2024-03-11T20:52:01.344963+0000","last_fullsized":"2024-03-11T20:52:01.344963+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:51:35.586120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709472+0000","last_change":"2024-03-11T20:51:11.729146+0000","last_active":"2024-03-11T20:52:00.709472+0000","last_peered":"2024-03-11T20:52:00.709472+0000","last_clean":"2024-03-11T20:52:00.709472+0000","last_became_active":"2024-03-11T20:51:11.728793+0000","last_became_peered":"2024-03-11T20:51:11.728793+0000","last_unstale":"2024-03-11T20:52:00.709472+0000","last_undegraded":"2024-03-11T20:52:00.709472+0000","last_fullsized":"2024-03-11T20:52:00.709472+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:01:45.885696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.708977+0000","last_change":"2024-03-11T20:51:13.735203+0000","last_active":"2024-03-11T20:52:00.708977+0000","last_peered":"2024-03-11T20:52:00.708977+0000","last_clean":"2024-03-11T20:52:00.708977+0000","last_became_active":"2024-03-11T20:51:13.734811+0000","last_became_peered":"2024-03-11T20:51:13.734811+0000","last_unstale":"2024-03-11T20:52:00.708977+0000","last_undegraded":"2024-03-11T20:52:00.708977+0000","last_fullsized":"2024-03-11T20:52:00.708977+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:13:07.932358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345181+0000","last_change":"2024-03-11T20:51:17.762609+0000","last_active":"2024-03-11T20:52:01.345181+0000","last_peered":"2024-03-11T20:52:01.345181+0000","last_clean":"2024-03-11T20:52:01.345181+0000","last_became_active":"2024-03-11T20:51:17.762310+0000","last_became_peered":"2024-03-11T20:51:17.762310+0000","last_unstale":"2024-03-11T20:52:01.345181+0000","last_undegraded":"2024-03-11T20:52:01.345181+0000","last_fullsized":"2024-03-11T20:52:01.345181+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:57:03.723372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788521+0000","last_change":"2024-03-11T20:51:15.757042+0000","last_active":"2024-03-11T20:52:16.788521+0000","last_peered":"2024-03-11T20:52:16.788521+0000","last_clean":"2024-03-11T20:52:16.788521+0000","last_became_active":"2024-03-11T20:51:15.756694+0000","last_became_peered":"2024-03-11T20:51:15.756694+0000","last_unstale":"2024-03-11T20:52:16.788521+0000","last_undegraded":"2024-03-11T20:52:16.788521+0000","last_fullsized":"2024-03-11T20:52:16.788521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:22:11.775053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828655+0000","last_change":"2024-03-11T20:51:11.733581+0000","last_active":"2024-03-11T20:52:00.828655+0000","last_peered":"2024-03-11T20:52:00.828655+0000","last_clean":"2024-03-11T20:52:00.828655+0000","last_became_active":"2024-03-11T20:51:11.729712+0000","last_became_peered":"2024-03-11T20:51:11.729712+0000","last_unstale":"2024-03-11T20:52:00.828655+0000","last_undegraded":"2024-03-11T20:52:00.828655+0000","last_fullsized":"2024-03-11T20:52:00.828655+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:46:06.746085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710375+0000","last_change":"2024-03-11T20:51:14.072014+0000","last_active":"2024-03-11T20:52:00.710375+0000","last_peered":"2024-03-11T20:52:00.710375+0000","last_clean":"2024-03-11T20:52:00.710375+0000","last_became_active":"2024-03-11T20:51:14.071760+0000","last_became_peered":"2024-03-11T20:51:14.071760+0000","last_unstale":"2024-03-11T20:52:00.710375+0000","last_undegraded":"2024-03-11T20:52:00.710375+0000","last_fullsized":"2024-03-11T20:52:00.710375+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:12:17.474059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710010+0000","last_change":"2024-03-11T20:51:17.757122+0000","last_active":"2024-03-11T20:52:00.710010+0000","last_peered":"2024-03-11T20:52:00.710010+0000","last_clean":"2024-03-11T20:52:00.710010+0000","last_became_active":"2024-03-11T20:51:17.756838+0000","last_became_peered":"2024-03-11T20:51:17.756838+0000","last_unstale":"2024-03-11T20:52:00.710010+0000","last_undegraded":"2024-03-11T20:52:00.710010+0000","last_fullsized":"2024-03-11T20:52:00.710010+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:05:59.758290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788673+0000","last_change":"2024-03-11T20:51:15.748370+0000","last_active":"2024-03-11T20:52:16.788673+0000","last_peered":"2024-03-11T20:52:16.788673+0000","last_clean":"2024-03-11T20:52:16.788673+0000","last_became_active":"2024-03-11T20:51:15.747526+0000","last_became_peered":"2024-03-11T20:51:15.747526+0000","last_unstale":"2024-03-11T20:52:16.788673+0000","last_undegraded":"2024-03-11T20:52:16.788673+0000","last_fullsized":"2024-03-11T20:52:16.788673+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:30:45.786798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344312+0000","last_change":"2024-03-11T20:51:11.725253+0000","last_active":"2024-03-11T20:52:01.344312+0000","last_peered":"2024-03-11T20:52:01.344312+0000","last_clean":"2024-03-11T20:52:01.344312+0000","last_became_active":"2024-03-11T20:51:11.724851+0000","last_became_peered":"2024-03-11T20:51:11.724851+0000","last_unstale":"2024-03-11T20:52:01.344312+0000","last_undegraded":"2024-03-11T20:52:01.344312+0000","last_fullsized":"2024-03-11T20:52:01.344312+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:20:23.723816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.822140+0000","last_change":"2024-03-11T20:51:14.073927+0000","last_active":"2024-03-11T20:52:00.822140+0000","last_peered":"2024-03-11T20:52:00.822140+0000","last_clean":"2024-03-11T20:52:00.822140+0000","last_became_active":"2024-03-11T20:51:14.073517+0000","last_became_peered":"2024-03-11T20:51:14.073517+0000","last_unstale":"2024-03-11T20:52:00.822140+0000","last_undegraded":"2024-03-11T20:52:00.822140+0000","last_fullsized":"2024-03-11T20:52:00.822140+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:47:38.822484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707742+0000","last_change":"2024-03-11T20:51:17.752450+0000","last_active":"2024-03-11T20:52:00.707742+0000","last_peered":"2024-03-11T20:52:00.707742+0000","last_clean":"2024-03-11T20:52:00.707742+0000","last_became_active":"2024-03-11T20:51:17.752115+0000","last_became_peered":"2024-03-11T20:51:17.752115+0000","last_unstale":"2024-03-11T20:52:00.707742+0000","last_undegraded":"2024-03-11T20:52:00.707742+0000","last_fullsized":"2024-03-11T20:52:00.707742+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:20:26.753447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.712412+0000","last_change":"2024-03-11T20:51:15.750052+0000","last_active":"2024-03-11T20:52:00.712412+0000","last_peered":"2024-03-11T20:52:00.712412+0000","last_clean":"2024-03-11T20:52:00.712412+0000","last_became_active":"2024-03-11T20:51:15.749562+0000","last_became_peered":"2024-03-11T20:51:15.749562+0000","last_unstale":"2024-03-11T20:52:00.712412+0000","last_undegraded":"2024-03-11T20:52:00.712412+0000","last_fullsized":"2024-03-11T20:52:00.712412+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:48:54.970489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828514+0000","last_change":"2024-03-11T20:51:11.730973+0000","last_active":"2024-03-11T20:52:00.828514+0000","last_peered":"2024-03-11T20:52:00.828514+0000","last_clean":"2024-03-11T20:52:00.828514+0000","last_became_active":"2024-03-11T20:51:11.729768+0000","last_became_peered":"2024-03-11T20:51:11.729768+0000","last_unstale":"2024-03-11T20:52:00.828514+0000","last_undegraded":"2024-03-11T20:52:00.828514+0000","last_fullsized":"2024-03-11T20:52:00.828514+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:23:38.724061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345408+0000","last_change":"2024-03-11T20:51:14.072688+0000","last_active":"2024-03-11T20:52:01.345408+0000","last_peered":"2024-03-11T20:52:01.345408+0000","last_clean":"2024-03-11T20:52:01.345408+0000","last_became_active":"2024-03-11T20:51:14.072368+0000","last_became_peered":"2024-03-11T20:51:14.072368+0000","last_unstale":"2024-03-11T20:52:01.345408+0000","last_undegraded":"2024-03-11T20:52:01.345408+0000","last_fullsized":"2024-03-11T20:52:01.345408+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:43:36.363151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828920+0000","last_change":"2024-03-11T20:51:17.888736+0000","last_active":"2024-03-11T20:52:00.828920+0000","last_peered":"2024-03-11T20:52:00.828920+0000","last_clean":"2024-03-11T20:52:00.828920+0000","last_became_active":"2024-03-11T20:51:17.888438+0000","last_became_peered":"2024-03-11T20:51:17.888438+0000","last_unstale":"2024-03-11T20:52:00.828920+0000","last_undegraded":"2024-03-11T20:52:00.828920+0000","last_fullsized":"2024-03-11T20:52:00.828920+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:57:52.210447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.822014+0000","last_change":"2024-03-11T20:51:15.752669+0000","last_active":"2024-03-11T20:52:00.822014+0000","last_peered":"2024-03-11T20:52:00.822014+0000","last_clean":"2024-03-11T20:52:00.822014+0000","last_became_active":"2024-03-11T20:51:15.752393+0000","last_became_peered":"2024-03-11T20:51:15.752393+0000","last_unstale":"2024-03-11T20:52:00.822014+0000","last_undegraded":"2024-03-11T20:52:00.822014+0000","last_fullsized":"2024-03-11T20:52:00.822014+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:33:01.024179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343647+0000","last_change":"2024-03-11T20:51:11.727464+0000","last_active":"2024-03-11T20:52:01.343647+0000","last_peered":"2024-03-11T20:52:01.343647+0000","last_clean":"2024-03-11T20:52:01.343647+0000","last_became_active":"2024-03-11T20:51:11.727159+0000","last_became_peered":"2024-03-11T20:51:11.727159+0000","last_unstale":"2024-03-11T20:52:01.343647+0000","last_undegraded":"2024-03-11T20:52:01.343647+0000","last_fullsized":"2024-03-11T20:52:01.343647+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:01:05.288045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":38,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.712514+0000","last_change":"2024-03-11T20:51:14.072241+0000","last_active":"2024-03-11T20:52:00.712514+0000","last_peered":"2024-03-11T20:52:00.712514+0000","last_clean":"2024-03-11T20:52:00.712514+0000","last_became_active":"2024-03-11T20:51:14.071882+0000","last_became_peered":"2024-03-11T20:51:14.071882+0000","last_unstale":"2024-03-11T20:52:00.712514+0000","last_undegraded":"2024-03-11T20:52:00.712514+0000","last_fullsized":"2024-03-11T20:52:00.712514+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:20:56.104843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821740+0000","last_change":"2024-03-11T20:51:17.889951+0000","last_active":"2024-03-11T20:52:00.821740+0000","last_peered":"2024-03-11T20:52:00.821740+0000","last_clean":"2024-03-11T20:52:00.821740+0000","last_became_active":"2024-03-11T20:51:17.889641+0000","last_became_peered":"2024-03-11T20:51:17.889641+0000","last_unstale":"2024-03-11T20:52:00.821740+0000","last_undegraded":"2024-03-11T20:52:00.821740+0000","last_fullsized":"2024-03-11T20:52:00.821740+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:42:24.770726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'3","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.829190+0000","last_change":"2024-03-11T20:51:11.731590+0000","last_active":"2024-03-11T20:52:00.829190+0000","last_peered":"2024-03-11T20:52:00.829190+0000","last_clean":"2024-03-11T20:52:00.829190+0000","last_became_active":"2024-03-11T20:51:11.731127+0000","last_became_peered":"2024-03-11T20:51:11.731127+0000","last_unstale":"2024-03-11T20:52:00.829190+0000","last_undegraded":"2024-03-11T20:52:00.829190+0000","last_fullsized":"2024-03-11T20:52:00.829190+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:40:10.498058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821452+0000","last_change":"2024-03-11T20:51:14.075015+0000","last_active":"2024-03-11T20:52:00.821452+0000","last_peered":"2024-03-11T20:52:00.821452+0000","last_clean":"2024-03-11T20:52:00.821452+0000","last_became_active":"2024-03-11T20:51:14.074664+0000","last_became_peered":"2024-03-11T20:51:14.074664+0000","last_unstale":"2024-03-11T20:52:00.821452+0000","last_undegraded":"2024-03-11T20:52:00.821452+0000","last_fullsized":"2024-03-11T20:52:00.821452+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:05.021441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:11.787437+0000","last_change":"2024-03-11T20:51:15.751443+0000","last_active":"2024-03-11T20:52:11.787437+0000","last_peered":"2024-03-11T20:52:11.787437+0000","last_clean":"2024-03-11T20:52:11.787437+0000","last_became_active":"2024-03-11T20:51:15.751142+0000","last_became_peered":"2024-03-11T20:51:15.751142+0000","last_unstale":"2024-03-11T20:52:11.787437+0000","last_undegraded":"2024-03-11T20:52:11.787437+0000","last_fullsized":"2024-03-11T20:52:11.787437+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:44.328862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710302+0000","last_change":"2024-03-11T20:51:17.757771+0000","last_active":"2024-03-11T20:52:00.710302+0000","last_peered":"2024-03-11T20:52:00.710302+0000","last_clean":"2024-03-11T20:52:00.710302+0000","last_became_active":"2024-03-11T20:51:17.757456+0000","last_became_peered":"2024-03-11T20:51:17.757456+0000","last_unstale":"2024-03-11T20:52:00.710302+0000","last_undegraded":"2024-03-11T20:52:00.710302+0000","last_fullsized":"2024-03-11T20:52:00.710302+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:24:17.674062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.829118+0000","last_change":"2024-03-11T20:51:11.730704+0000","last_active":"2024-03-11T20:52:00.829118+0000","last_peered":"2024-03-11T20:52:00.829118+0000","last_clean":"2024-03-11T20:52:00.829118+0000","last_became_active":"2024-03-11T20:51:11.729468+0000","last_became_peered":"2024-03-11T20:51:11.729468+0000","last_unstale":"2024-03-11T20:52:00.829118+0000","last_undegraded":"2024-03-11T20:52:00.829118+0000","last_fullsized":"2024-03-11T20:52:00.829118+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:00:58.165332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":70,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707770+0000","last_change":"2024-03-11T20:51:31.762808+0000","last_active":"2024-03-11T20:52:00.707770+0000","last_peered":"2024-03-11T20:52:00.707770+0000","last_clean":"2024-03-11T20:52:00.707770+0000","last_became_active":"2024-03-11T20:51:31.756369+0000","last_became_peered":"2024-03-11T20:51:31.756369+0000","last_unstale":"2024-03-11T20:52:00.707770+0000","last_undegraded":"2024-03-11T20:52:00.707770+0000","last_fullsized":"2024-03-11T20:52:00.707770+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:14:01.083562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.706815+0000","last_change":"2024-03-11T20:51:15.745193+0000","last_active":"2024-03-11T20:52:00.706815+0000","last_peered":"2024-03-11T20:52:00.706815+0000","last_clean":"2024-03-11T20:52:00.706815+0000","last_became_active":"2024-03-11T20:51:15.744733+0000","last_became_peered":"2024-03-11T20:51:15.744733+0000","last_unstale":"2024-03-11T20:52:00.706815+0000","last_undegraded":"2024-03-11T20:52:00.706815+0000","last_fullsized":"2024-03-11T20:52:00.706815+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:03:33.739180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828341+0000","last_change":"2024-03-11T20:51:17.759356+0000","last_active":"2024-03-11T20:52:00.828341+0000","last_peered":"2024-03-11T20:52:00.828341+0000","last_clean":"2024-03-11T20:52:00.828341+0000","last_became_active":"2024-03-11T20:51:17.759097+0000","last_became_peered":"2024-03-11T20:51:17.759097+0000","last_unstale":"2024-03-11T20:52:00.828341+0000","last_undegraded":"2024-03-11T20:52:00.828341+0000","last_fullsized":"2024-03-11T20:52:00.828341+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:50:15.438548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707495+0000","last_change":"2024-03-11T20:51:15.750672+0000","last_active":"2024-03-11T20:52:00.707495+0000","last_peered":"2024-03-11T20:52:00.707495+0000","last_clean":"2024-03-11T20:52:00.707495+0000","last_became_active":"2024-03-11T20:51:15.750295+0000","last_became_peered":"2024-03-11T20:51:15.750295+0000","last_unstale":"2024-03-11T20:52:00.707495+0000","last_undegraded":"2024-03-11T20:52:00.707495+0000","last_fullsized":"2024-03-11T20:52:00.707495+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:42:38.604383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.829039+0000","last_change":"2024-03-11T20:51:11.727910+0000","last_active":"2024-03-11T20:52:00.829039+0000","last_peered":"2024-03-11T20:52:00.829039+0000","last_clean":"2024-03-11T20:52:00.829039+0000","last_became_active":"2024-03-11T20:51:11.727576+0000","last_became_peered":"2024-03-11T20:51:11.727576+0000","last_unstale":"2024-03-11T20:52:00.829039+0000","last_undegraded":"2024-03-11T20:52:00.829039+0000","last_fullsized":"2024-03-11T20:52:00.829039+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:41.460100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343837+0000","last_change":"2024-03-11T20:51:14.073803+0000","last_active":"2024-03-11T20:52:01.343837+0000","last_peered":"2024-03-11T20:52:01.343837+0000","last_clean":"2024-03-11T20:52:01.343837+0000","last_became_active":"2024-03-11T20:51:14.073462+0000","last_became_peered":"2024-03-11T20:51:14.073462+0000","last_unstale":"2024-03-11T20:52:01.343837+0000","last_undegraded":"2024-03-11T20:52:01.343837+0000","last_fullsized":"2024-03-11T20:52:01.343837+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:27:19.898543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343993+0000","last_change":"2024-03-11T20:51:17.760664+0000","last_active":"2024-03-11T20:52:01.343993+0000","last_peered":"2024-03-11T20:52:01.343993+0000","last_clean":"2024-03-11T20:52:01.343993+0000","last_became_active":"2024-03-11T20:51:17.760350+0000","last_became_peered":"2024-03-11T20:51:17.760350+0000","last_unstale":"2024-03-11T20:52:01.343993+0000","last_undegraded":"2024-03-11T20:52:01.343993+0000","last_fullsized":"2024-03-11T20:52:01.343993+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:34:28.076159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707280+0000","last_change":"2024-03-11T20:51:15.746397+0000","last_active":"2024-03-11T20:52:00.707280+0000","last_peered":"2024-03-11T20:52:00.707280+0000","last_clean":"2024-03-11T20:52:00.707280+0000","last_became_active":"2024-03-11T20:51:15.746074+0000","last_became_peered":"2024-03-11T20:51:15.746074+0000","last_unstale":"2024-03-11T20:52:00.707280+0000","last_undegraded":"2024-03-11T20:52:00.707280+0000","last_fullsized":"2024-03-11T20:52:00.707280+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:07:12.110877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821823+0000","last_change":"2024-03-11T20:51:11.726209+0000","last_active":"2024-03-11T20:52:00.821823+0000","last_peered":"2024-03-11T20:52:00.821823+0000","last_clean":"2024-03-11T20:52:00.821823+0000","last_became_active":"2024-03-11T20:51:11.725872+0000","last_became_peered":"2024-03-11T20:51:11.725872+0000","last_unstale":"2024-03-11T20:52:00.821823+0000","last_undegraded":"2024-03-11T20:52:00.821823+0000","last_fullsized":"2024-03-11T20:52:00.821823+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:34:31.051009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344661+0000","last_change":"2024-03-11T20:51:13.735310+0000","last_active":"2024-03-11T20:52:01.344661+0000","last_peered":"2024-03-11T20:52:01.344661+0000","last_clean":"2024-03-11T20:52:01.344661+0000","last_became_active":"2024-03-11T20:51:13.734663+0000","last_became_peered":"2024-03-11T20:51:13.734663+0000","last_unstale":"2024-03-11T20:52:01.344661+0000","last_undegraded":"2024-03-11T20:52:01.344661+0000","last_fullsized":"2024-03-11T20:52:01.344661+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:01:57.099493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710874+0000","last_change":"2024-03-11T20:51:17.756929+0000","last_active":"2024-03-11T20:52:00.710874+0000","last_peered":"2024-03-11T20:52:00.710874+0000","last_clean":"2024-03-11T20:52:00.710874+0000","last_became_active":"2024-03-11T20:51:17.756629+0000","last_became_peered":"2024-03-11T20:51:17.756629+0000","last_unstale":"2024-03-11T20:52:00.710874+0000","last_undegraded":"2024-03-11T20:52:00.710874+0000","last_fullsized":"2024-03-11T20:52:00.710874+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:49:01.858696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821578+0000","last_change":"2024-03-11T20:51:14.733852+0000","last_active":"2024-03-11T20:52:00.821578+0000","last_peered":"2024-03-11T20:52:00.821578+0000","last_clean":"2024-03-11T20:52:00.821578+0000","last_became_active":"2024-03-11T20:51:10.707493+0000","last_became_peered":"2024-03-11T20:51:10.707493+0000","last_unstale":"2024-03-11T20:52:00.821578+0000","last_undegraded":"2024-03-11T20:52:00.821578+0000","last_fullsized":"2024-03-11T20:52:00.821578+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:09.689152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:09.689152+0000","last_clean_scrub_stamp":"2024-03-11T20:51:09.689152+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:06:20.986387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001974848,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.708598+0000","last_change":"2024-03-11T20:51:11.728606+0000","last_active":"2024-03-11T20:52:00.708598+0000","last_peered":"2024-03-11T20:52:00.708598+0000","last_clean":"2024-03-11T20:52:00.708598+0000","last_became_active":"2024-03-11T20:51:11.728289+0000","last_became_peered":"2024-03-11T20:51:11.728289+0000","last_unstale":"2024-03-11T20:52:00.708598+0000","last_undegraded":"2024-03-11T20:52:00.708598+0000","last_fullsized":"2024-03-11T20:52:00.708598+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:30:44.896019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":92,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788341+0000","last_change":"2024-03-11T20:51:13.739087+0000","last_active":"2024-03-11T20:52:16.788341+0000","last_peered":"2024-03-11T20:52:16.788341+0000","last_clean":"2024-03-11T20:52:16.788341+0000","last_became_active":"2024-03-11T20:51:13.738746+0000","last_became_peered":"2024-03-11T20:51:13.738746+0000","last_unstale":"2024-03-11T20:52:16.788341+0000","last_undegraded":"2024-03-11T20:52:16.788341+0000","last_fullsized":"2024-03-11T20:52:16.788341+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:57:23.938142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344600+0000","last_change":"2024-03-11T20:51:15.747401+0000","last_active":"2024-03-11T20:52:01.344600+0000","last_peered":"2024-03-11T20:52:01.344600+0000","last_clean":"2024-03-11T20:52:01.344600+0000","last_became_active":"2024-03-11T20:51:15.747010+0000","last_became_peered":"2024-03-11T20:51:15.747010+0000","last_unstale":"2024-03-11T20:52:01.344600+0000","last_undegraded":"2024-03-11T20:52:01.344600+0000","last_fullsized":"2024-03-11T20:52:01.344600+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:42:50.389808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.829096+0000","last_change":"2024-03-11T20:51:17.890420+0000","last_active":"2024-03-11T20:52:00.829096+0000","last_peered":"2024-03-11T20:52:00.829096+0000","last_clean":"2024-03-11T20:52:00.829096+0000","last_became_active":"2024-03-11T20:51:17.889919+0000","last_became_peered":"2024-03-11T20:51:17.889919+0000","last_unstale":"2024-03-11T20:52:00.829096+0000","last_undegraded":"2024-03-11T20:52:00.829096+0000","last_fullsized":"2024-03-11T20:52:00.829096+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:55:01.662146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344764+0000","last_change":"2024-03-11T20:51:11.723344+0000","last_active":"2024-03-11T20:52:01.344764+0000","last_peered":"2024-03-11T20:52:01.344764+0000","last_clean":"2024-03-11T20:52:01.344764+0000","last_became_active":"2024-03-11T20:51:11.722921+0000","last_became_peered":"2024-03-11T20:51:11.722921+0000","last_unstale":"2024-03-11T20:52:01.344764+0000","last_undegraded":"2024-03-11T20:52:01.344764+0000","last_fullsized":"2024-03-11T20:52:01.344764+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:30:48.721567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707070+0000","last_change":"2024-03-11T20:51:14.728335+0000","last_active":"2024-03-11T20:52:00.707070+0000","last_peered":"2024-03-11T20:52:00.707070+0000","last_clean":"2024-03-11T20:52:00.707070+0000","last_became_active":"2024-03-11T20:51:10.709107+0000","last_became_peered":"2024-03-11T20:51:10.709107+0000","last_unstale":"2024-03-11T20:52:00.707070+0000","last_undegraded":"2024-03-11T20:52:00.707070+0000","last_fullsized":"2024-03-11T20:52:00.707070+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:09.689152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:09.689152+0000","last_clean_scrub_stamp":"2024-03-11T20:51:09.689152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:53:21.681104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054735999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":56,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345761+0000","last_change":"2024-03-11T20:51:13.736954+0000","last_active":"2024-03-11T20:52:01.345761+0000","last_peered":"2024-03-11T20:52:01.345761+0000","last_clean":"2024-03-11T20:52:01.345761+0000","last_became_active":"2024-03-11T20:51:13.736669+0000","last_became_peered":"2024-03-11T20:51:13.736669+0000","last_unstale":"2024-03-11T20:52:01.345761+0000","last_undegraded":"2024-03-11T20:52:01.345761+0000","last_fullsized":"2024-03-11T20:52:01.345761+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:55:46.337370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.708120+0000","last_change":"2024-03-11T20:51:15.751380+0000","last_active":"2024-03-11T20:52:00.708120+0000","last_peered":"2024-03-11T20:52:00.708120+0000","last_clean":"2024-03-11T20:52:00.708120+0000","last_became_active":"2024-03-11T20:51:15.751088+0000","last_became_peered":"2024-03-11T20:51:15.751088+0000","last_unstale":"2024-03-11T20:52:00.708120+0000","last_undegraded":"2024-03-11T20:52:00.708120+0000","last_fullsized":"2024-03-11T20:52:00.708120+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:35:14.185900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.822112+0000","last_change":"2024-03-11T20:51:17.754996+0000","last_active":"2024-03-11T20:52:00.822112+0000","last_peered":"2024-03-11T20:52:00.822112+0000","last_clean":"2024-03-11T20:52:00.822112+0000","last_became_active":"2024-03-11T20:51:17.754690+0000","last_became_peered":"2024-03-11T20:51:17.754690+0000","last_unstale":"2024-03-11T20:52:00.822112+0000","last_undegraded":"2024-03-11T20:52:00.822112+0000","last_fullsized":"2024-03-11T20:52:00.822112+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:48:06.180451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"72'128","reported_seq":278,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:02.806506+0000","last_change":"2024-03-11T20:51:01.493203+0000","last_active":"2024-03-11T20:52:02.806506+0000","last_peered":"2024-03-11T20:52:02.806506+0000","last_clean":"2024-03-11T20:52:02.806506+0000","last_became_active":"2024-03-11T20:51:01.492723+0000","last_became_peered":"2024-03-11T20:51:01.492723+0000","last_unstale":"2024-03-11T20:52:02.806506+0000","last_undegraded":"2024-03-11T20:52:02.806506+0000","last_fullsized":"2024-03-11T20:52:02.806506+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:48:50.017181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:48:50.017181+0000","last_clean_scrub_stamp":"2024-03-11T20:48:50.017181+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:38.769263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"61'1","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.711847+0000","last_change":"2024-03-11T20:51:11.723644+0000","last_active":"2024-03-11T20:52:00.711847+0000","last_peered":"2024-03-11T20:52:00.711847+0000","last_clean":"2024-03-11T20:52:00.711847+0000","last_became_active":"2024-03-11T20:51:11.723310+0000","last_became_peered":"2024-03-11T20:51:11.723310+0000","last_unstale":"2024-03-11T20:52:00.711847+0000","last_undegraded":"2024-03-11T20:52:00.711847+0000","last_fullsized":"2024-03-11T20:52:00.711847+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:11:33.944833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":65,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:11.787671+0000","last_change":"2024-03-11T20:51:14.072215+0000","last_active":"2024-03-11T20:52:11.787671+0000","last_peered":"2024-03-11T20:52:11.787671+0000","last_clean":"2024-03-11T20:52:11.787671+0000","last_became_active":"2024-03-11T20:51:14.071826+0000","last_became_peered":"2024-03-11T20:51:14.071826+0000","last_unstale":"2024-03-11T20:52:11.787671+0000","last_undegraded":"2024-03-11T20:52:11.787671+0000","last_fullsized":"2024-03-11T20:52:11.787671+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:31:42.729179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828782+0000","last_change":"2024-03-11T20:51:15.748126+0000","last_active":"2024-03-11T20:52:00.828782+0000","last_peered":"2024-03-11T20:52:00.828782+0000","last_clean":"2024-03-11T20:52:00.828782+0000","last_became_active":"2024-03-11T20:51:15.747840+0000","last_became_peered":"2024-03-11T20:51:15.747840+0000","last_unstale":"2024-03-11T20:52:00.828782+0000","last_undegraded":"2024-03-11T20:52:00.828782+0000","last_fullsized":"2024-03-11T20:52:00.828782+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:14:29.960588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345140+0000","last_change":"2024-03-11T20:51:17.757756+0000","last_active":"2024-03-11T20:52:01.345140+0000","last_peered":"2024-03-11T20:52:01.345140+0000","last_clean":"2024-03-11T20:52:01.345140+0000","last_became_active":"2024-03-11T20:51:17.757474+0000","last_became_peered":"2024-03-11T20:51:17.757474+0000","last_unstale":"2024-03-11T20:52:01.345140+0000","last_undegraded":"2024-03-11T20:52:01.345140+0000","last_fullsized":"2024-03-11T20:52:01.345140+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:50:07.033497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344214+0000","last_change":"2024-03-11T20:51:11.729185+0000","last_active":"2024-03-11T20:52:01.344214+0000","last_peered":"2024-03-11T20:52:01.344214+0000","last_clean":"2024-03-11T20:52:01.344214+0000","last_became_active":"2024-03-11T20:51:11.728866+0000","last_became_peered":"2024-03-11T20:51:11.728866+0000","last_unstale":"2024-03-11T20:52:01.344214+0000","last_undegraded":"2024-03-11T20:52:01.344214+0000","last_fullsized":"2024-03-11T20:52:01.344214+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:38:44.383310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":109,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:20.229404+0000","last_change":"2024-03-11T20:51:14.734085+0000","last_active":"2024-03-11T20:52:20.229404+0000","last_peered":"2024-03-11T20:52:20.229404+0000","last_clean":"2024-03-11T20:52:20.229404+0000","last_became_active":"2024-03-11T20:51:10.707227+0000","last_became_peered":"2024-03-11T20:51:10.707227+0000","last_unstale":"2024-03-11T20:52:20.229404+0000","last_undegraded":"2024-03-11T20:52:20.229404+0000","last_fullsized":"2024-03-11T20:52:20.229404+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:09.689152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:09.689152+0000","last_clean_scrub_stamp":"2024-03-11T20:51:09.689152+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:05:07.734778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046710399999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":64,"num_read_kb":59,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":47,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345181+0000","last_change":"2024-03-11T20:51:13.735009+0000","last_active":"2024-03-11T20:52:01.345181+0000","last_peered":"2024-03-11T20:52:01.345181+0000","last_clean":"2024-03-11T20:52:01.345181+0000","last_became_active":"2024-03-11T20:51:13.734654+0000","last_became_peered":"2024-03-11T20:51:13.734654+0000","last_unstale":"2024-03-11T20:52:01.345181+0000","last_undegraded":"2024-03-11T20:52:01.345181+0000","last_fullsized":"2024-03-11T20:52:01.345181+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:55:49.732038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.822235+0000","last_change":"2024-03-11T20:51:15.756612+0000","last_active":"2024-03-11T20:52:00.822235+0000","last_peered":"2024-03-11T20:52:00.822235+0000","last_clean":"2024-03-11T20:52:00.822235+0000","last_became_active":"2024-03-11T20:51:15.756110+0000","last_became_peered":"2024-03-11T20:51:15.756110+0000","last_unstale":"2024-03-11T20:52:00.822235+0000","last_undegraded":"2024-03-11T20:52:00.822235+0000","last_fullsized":"2024-03-11T20:52:00.822235+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:54.185806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344572+0000","last_change":"2024-03-11T20:51:17.888814+0000","last_active":"2024-03-11T20:52:01.344572+0000","last_peered":"2024-03-11T20:52:01.344572+0000","last_clean":"2024-03-11T20:52:01.344572+0000","last_became_active":"2024-03-11T20:51:17.888460+0000","last_became_peered":"2024-03-11T20:51:17.888460+0000","last_unstale":"2024-03-11T20:52:01.344572+0000","last_undegraded":"2024-03-11T20:52:01.344572+0000","last_fullsized":"2024-03-11T20:52:01.344572+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:59:29.851934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345345+0000","last_change":"2024-03-11T20:51:11.714853+0000","last_active":"2024-03-11T20:52:01.345345+0000","last_peered":"2024-03-11T20:52:01.345345+0000","last_clean":"2024-03-11T20:52:01.345345+0000","last_became_active":"2024-03-11T20:51:11.714501+0000","last_became_peered":"2024-03-11T20:51:11.714501+0000","last_unstale":"2024-03-11T20:52:01.345345+0000","last_undegraded":"2024-03-11T20:52:01.345345+0000","last_fullsized":"2024-03-11T20:52:01.345345+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:37.956357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":65,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344919+0000","last_change":"2024-03-11T20:51:14.071827+0000","last_active":"2024-03-11T20:52:01.344919+0000","last_peered":"2024-03-11T20:52:01.344919+0000","last_clean":"2024-03-11T20:52:01.344919+0000","last_became_active":"2024-03-11T20:51:14.071470+0000","last_became_peered":"2024-03-11T20:51:14.071470+0000","last_unstale":"2024-03-11T20:52:01.344919+0000","last_undegraded":"2024-03-11T20:52:01.344919+0000","last_fullsized":"2024-03-11T20:52:01.344919+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:02.382794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343923+0000","last_change":"2024-03-11T20:51:15.751107+0000","last_active":"2024-03-11T20:52:01.343923+0000","last_peered":"2024-03-11T20:52:01.343923+0000","last_clean":"2024-03-11T20:52:01.343923+0000","last_became_active":"2024-03-11T20:51:15.750808+0000","last_became_peered":"2024-03-11T20:51:15.750808+0000","last_unstale":"2024-03-11T20:52:01.343923+0000","last_undegraded":"2024-03-11T20:52:01.343923+0000","last_fullsized":"2024-03-11T20:52:01.343923+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:55:22.651002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343760+0000","last_change":"2024-03-11T20:51:17.762486+0000","last_active":"2024-03-11T20:52:01.343760+0000","last_peered":"2024-03-11T20:52:01.343760+0000","last_clean":"2024-03-11T20:52:01.343760+0000","last_became_active":"2024-03-11T20:51:17.762153+0000","last_became_peered":"2024-03-11T20:51:17.762153+0000","last_unstale":"2024-03-11T20:52:01.343760+0000","last_undegraded":"2024-03-11T20:52:01.343760+0000","last_fullsized":"2024-03-11T20:52:01.343760+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:38.439624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.822018+0000","last_change":"2024-03-11T20:51:11.723320+0000","last_active":"2024-03-11T20:52:00.822018+0000","last_peered":"2024-03-11T20:52:00.822018+0000","last_clean":"2024-03-11T20:52:00.822018+0000","last_became_active":"2024-03-11T20:51:11.722992+0000","last_became_peered":"2024-03-11T20:51:11.722992+0000","last_unstale":"2024-03-11T20:52:00.822018+0000","last_undegraded":"2024-03-11T20:52:00.822018+0000","last_fullsized":"2024-03-11T20:52:00.822018+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:16.876429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.706972+0000","last_change":"2024-03-11T20:51:31.763694+0000","last_active":"2024-03-11T20:52:00.706972+0000","last_peered":"2024-03-11T20:52:00.706972+0000","last_clean":"2024-03-11T20:52:00.706972+0000","last_became_active":"2024-03-11T20:51:31.756525+0000","last_became_peered":"2024-03-11T20:51:31.756525+0000","last_unstale":"2024-03-11T20:52:00.706972+0000","last_undegraded":"2024-03-11T20:52:00.706972+0000","last_fullsized":"2024-03-11T20:52:00.706972+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:09:45.138476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788331+0000","last_change":"2024-03-11T20:51:15.753138+0000","last_active":"2024-03-11T20:52:16.788331+0000","last_peered":"2024-03-11T20:52:16.788331+0000","last_clean":"2024-03-11T20:52:16.788331+0000","last_became_active":"2024-03-11T20:51:15.752789+0000","last_became_peered":"2024-03-11T20:51:15.752789+0000","last_unstale":"2024-03-11T20:52:16.788331+0000","last_undegraded":"2024-03-11T20:52:16.788331+0000","last_fullsized":"2024-03-11T20:52:16.788331+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:49:48.857603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828889+0000","last_change":"2024-03-11T20:51:17.890460+0000","last_active":"2024-03-11T20:52:00.828889+0000","last_peered":"2024-03-11T20:52:00.828889+0000","last_clean":"2024-03-11T20:52:00.828889+0000","last_became_active":"2024-03-11T20:51:17.890176+0000","last_became_peered":"2024-03-11T20:51:17.890176+0000","last_unstale":"2024-03-11T20:52:00.828889+0000","last_undegraded":"2024-03-11T20:52:00.828889+0000","last_fullsized":"2024-03-11T20:52:00.828889+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:14:14.090671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345293+0000","last_change":"2024-03-11T20:51:11.729566+0000","last_active":"2024-03-11T20:52:01.345293+0000","last_peered":"2024-03-11T20:52:01.345293+0000","last_clean":"2024-03-11T20:52:01.345293+0000","last_became_active":"2024-03-11T20:51:11.728869+0000","last_became_peered":"2024-03-11T20:51:11.728869+0000","last_unstale":"2024-03-11T20:52:01.345293+0000","last_undegraded":"2024-03-11T20:52:01.345293+0000","last_fullsized":"2024-03-11T20:52:01.345293+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:49:57.454894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707567+0000","last_change":"2024-03-11T20:51:13.730891+0000","last_active":"2024-03-11T20:52:00.707567+0000","last_peered":"2024-03-11T20:52:00.707567+0000","last_clean":"2024-03-11T20:52:00.707567+0000","last_became_active":"2024-03-11T20:51:13.730578+0000","last_became_peered":"2024-03-11T20:51:13.730578+0000","last_unstale":"2024-03-11T20:52:00.707567+0000","last_undegraded":"2024-03-11T20:52:00.707567+0000","last_fullsized":"2024-03-11T20:52:00.707567+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:57:59.848193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788329+0000","last_change":"2024-03-11T20:51:15.751998+0000","last_active":"2024-03-11T20:52:16.788329+0000","last_peered":"2024-03-11T20:52:16.788329+0000","last_clean":"2024-03-11T20:52:16.788329+0000","last_became_active":"2024-03-11T20:51:15.751637+0000","last_became_peered":"2024-03-11T20:51:15.751637+0000","last_unstale":"2024-03-11T20:52:16.788329+0000","last_undegraded":"2024-03-11T20:52:16.788329+0000","last_fullsized":"2024-03-11T20:52:16.788329+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:03.303229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707490+0000","last_change":"2024-03-11T20:51:17.757133+0000","last_active":"2024-03-11T20:52:00.707490+0000","last_peered":"2024-03-11T20:52:00.707490+0000","last_clean":"2024-03-11T20:52:00.707490+0000","last_became_active":"2024-03-11T20:51:17.756830+0000","last_became_peered":"2024-03-11T20:51:17.756830+0000","last_unstale":"2024-03-11T20:52:00.707490+0000","last_undegraded":"2024-03-11T20:52:00.707490+0000","last_fullsized":"2024-03-11T20:52:00.707490+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:32:16.808215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.711380+0000","last_change":"2024-03-11T20:51:11.725766+0000","last_active":"2024-03-11T20:52:00.711380+0000","last_peered":"2024-03-11T20:52:00.711380+0000","last_clean":"2024-03-11T20:52:00.711380+0000","last_became_active":"2024-03-11T20:51:11.725376+0000","last_became_peered":"2024-03-11T20:51:11.725376+0000","last_unstale":"2024-03-11T20:52:00.711380+0000","last_undegraded":"2024-03-11T20:52:00.711380+0000","last_fullsized":"2024-03-11T20:52:00.711380+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:29:50.920405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":61,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.711298+0000","last_change":"2024-03-11T20:51:14.072597+0000","last_active":"2024-03-11T20:52:00.711298+0000","last_peered":"2024-03-11T20:52:00.711298+0000","last_clean":"2024-03-11T20:52:00.711298+0000","last_became_active":"2024-03-11T20:51:14.072354+0000","last_became_peered":"2024-03-11T20:51:14.072354+0000","last_unstale":"2024-03-11T20:52:00.711298+0000","last_undegraded":"2024-03-11T20:52:00.711298+0000","last_fullsized":"2024-03-11T20:52:00.711298+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:53.066834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707326+0000","last_change":"2024-03-11T20:51:15.745710+0000","last_active":"2024-03-11T20:52:00.707326+0000","last_peered":"2024-03-11T20:52:00.707326+0000","last_clean":"2024-03-11T20:52:00.707326+0000","last_became_active":"2024-03-11T20:51:15.745425+0000","last_became_peered":"2024-03-11T20:51:15.745425+0000","last_unstale":"2024-03-11T20:52:00.707326+0000","last_undegraded":"2024-03-11T20:52:00.707326+0000","last_fullsized":"2024-03-11T20:52:00.707326+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:13:41.155899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821825+0000","last_change":"2024-03-11T20:51:17.758624+0000","last_active":"2024-03-11T20:52:00.821825+0000","last_peered":"2024-03-11T20:52:00.821825+0000","last_clean":"2024-03-11T20:52:00.821825+0000","last_became_active":"2024-03-11T20:51:17.758205+0000","last_became_peered":"2024-03-11T20:51:17.758205+0000","last_unstale":"2024-03-11T20:52:00.821825+0000","last_undegraded":"2024-03-11T20:52:00.821825+0000","last_fullsized":"2024-03-11T20:52:00.821825+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:30:21.138507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788501+0000","last_change":"2024-03-11T20:51:15.752710+0000","last_active":"2024-03-11T20:52:16.788501+0000","last_peered":"2024-03-11T20:52:16.788501+0000","last_clean":"2024-03-11T20:52:16.788501+0000","last_became_active":"2024-03-11T20:51:15.752429+0000","last_became_peered":"2024-03-11T20:51:15.752429+0000","last_unstale":"2024-03-11T20:52:16.788501+0000","last_undegraded":"2024-03-11T20:52:16.788501+0000","last_fullsized":"2024-03-11T20:52:16.788501+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:41:42.487209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709631+0000","last_change":"2024-03-11T20:51:11.722958+0000","last_active":"2024-03-11T20:52:00.709631+0000","last_peered":"2024-03-11T20:52:00.709631+0000","last_clean":"2024-03-11T20:52:00.709631+0000","last_became_active":"2024-03-11T20:51:11.722655+0000","last_became_peered":"2024-03-11T20:51:11.722655+0000","last_unstale":"2024-03-11T20:52:00.709631+0000","last_undegraded":"2024-03-11T20:52:00.709631+0000","last_fullsized":"2024-03-11T20:52:00.709631+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:16:42.699212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345943+0000","last_change":"2024-03-11T20:51:13.738088+0000","last_active":"2024-03-11T20:52:01.345943+0000","last_peered":"2024-03-11T20:52:01.345943+0000","last_clean":"2024-03-11T20:52:01.345943+0000","last_became_active":"2024-03-11T20:51:13.737794+0000","last_became_peered":"2024-03-11T20:51:13.737794+0000","last_unstale":"2024-03-11T20:52:01.345943+0000","last_undegraded":"2024-03-11T20:52:01.345943+0000","last_fullsized":"2024-03-11T20:52:01.345943+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:58:23.606555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710278+0000","last_change":"2024-03-11T20:51:17.888681+0000","last_active":"2024-03-11T20:52:00.710278+0000","last_peered":"2024-03-11T20:52:00.710278+0000","last_clean":"2024-03-11T20:52:00.710278+0000","last_became_active":"2024-03-11T20:51:17.888347+0000","last_became_peered":"2024-03-11T20:51:17.888347+0000","last_unstale":"2024-03-11T20:52:00.710278+0000","last_undegraded":"2024-03-11T20:52:00.710278+0000","last_fullsized":"2024-03-11T20:52:00.710278+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:06:28.451795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.822230+0000","last_change":"2024-03-11T20:51:15.753932+0000","last_active":"2024-03-11T20:52:00.822230+0000","last_peered":"2024-03-11T20:52:00.822230+0000","last_clean":"2024-03-11T20:52:00.822230+0000","last_became_active":"2024-03-11T20:51:15.753549+0000","last_became_peered":"2024-03-11T20:51:15.753549+0000","last_unstale":"2024-03-11T20:52:00.822230+0000","last_undegraded":"2024-03-11T20:52:00.822230+0000","last_fullsized":"2024-03-11T20:52:00.822230+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:37:10.766409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707404+0000","last_change":"2024-03-11T20:51:11.728497+0000","last_active":"2024-03-11T20:52:00.707404+0000","last_peered":"2024-03-11T20:52:00.707404+0000","last_clean":"2024-03-11T20:52:00.707404+0000","last_became_active":"2024-03-11T20:51:11.727489+0000","last_became_peered":"2024-03-11T20:51:11.727489+0000","last_unstale":"2024-03-11T20:52:00.707404+0000","last_undegraded":"2024-03-11T20:52:00.707404+0000","last_fullsized":"2024-03-11T20:52:00.707404+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:39.400474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709232+0000","last_change":"2024-03-11T20:51:13.735720+0000","last_active":"2024-03-11T20:52:00.709232+0000","last_peered":"2024-03-11T20:52:00.709232+0000","last_clean":"2024-03-11T20:52:00.709232+0000","last_became_active":"2024-03-11T20:51:13.734446+0000","last_became_peered":"2024-03-11T20:51:13.734446+0000","last_unstale":"2024-03-11T20:52:00.709232+0000","last_undegraded":"2024-03-11T20:52:00.709232+0000","last_fullsized":"2024-03-11T20:52:00.709232+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:28:44.865346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.708036+0000","last_change":"2024-03-11T20:51:17.755796+0000","last_active":"2024-03-11T20:52:00.708036+0000","last_peered":"2024-03-11T20:52:00.708036+0000","last_clean":"2024-03-11T20:52:00.708036+0000","last_became_active":"2024-03-11T20:51:17.755488+0000","last_became_peered":"2024-03-11T20:51:17.755488+0000","last_unstale":"2024-03-11T20:52:00.708036+0000","last_undegraded":"2024-03-11T20:52:00.708036+0000","last_fullsized":"2024-03-11T20:52:00.708036+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:58:24.604313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345687+0000","last_change":"2024-03-11T20:51:15.743301+0000","last_active":"2024-03-11T20:52:01.345687+0000","last_peered":"2024-03-11T20:52:01.345687+0000","last_clean":"2024-03-11T20:52:01.345687+0000","last_became_active":"2024-03-11T20:51:15.742824+0000","last_became_peered":"2024-03-11T20:51:15.742824+0000","last_unstale":"2024-03-11T20:52:01.345687+0000","last_undegraded":"2024-03-11T20:52:01.345687+0000","last_fullsized":"2024-03-11T20:52:01.345687+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:00.074718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343986+0000","last_change":"2024-03-11T20:51:11.731540+0000","last_active":"2024-03-11T20:52:01.343986+0000","last_peered":"2024-03-11T20:52:01.343986+0000","last_clean":"2024-03-11T20:52:01.343986+0000","last_became_active":"2024-03-11T20:51:11.731190+0000","last_became_peered":"2024-03-11T20:51:11.731190+0000","last_unstale":"2024-03-11T20:52:01.343986+0000","last_undegraded":"2024-03-11T20:52:01.343986+0000","last_fullsized":"2024-03-11T20:52:01.343986+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:32:05.461020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.708999+0000","last_change":"2024-03-11T20:51:13.736153+0000","last_active":"2024-03-11T20:52:00.708999+0000","last_peered":"2024-03-11T20:52:00.708999+0000","last_clean":"2024-03-11T20:52:00.708999+0000","last_became_active":"2024-03-11T20:51:13.734398+0000","last_became_peered":"2024-03-11T20:51:13.734398+0000","last_unstale":"2024-03-11T20:52:00.708999+0000","last_undegraded":"2024-03-11T20:52:00.708999+0000","last_fullsized":"2024-03-11T20:52:00.708999+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:30:13.152842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709585+0000","last_change":"2024-03-11T20:51:17.756970+0000","last_active":"2024-03-11T20:52:00.709585+0000","last_peered":"2024-03-11T20:52:00.709585+0000","last_clean":"2024-03-11T20:52:00.709585+0000","last_became_active":"2024-03-11T20:51:17.756656+0000","last_became_peered":"2024-03-11T20:51:17.756656+0000","last_unstale":"2024-03-11T20:52:00.709585+0000","last_undegraded":"2024-03-11T20:52:00.709585+0000","last_fullsized":"2024-03-11T20:52:00.709585+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:30:16.480717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788338+0000","last_change":"2024-03-11T20:51:15.752251+0000","last_active":"2024-03-11T20:52:16.788338+0000","last_peered":"2024-03-11T20:52:16.788338+0000","last_clean":"2024-03-11T20:52:16.788338+0000","last_became_active":"2024-03-11T20:51:15.751954+0000","last_became_peered":"2024-03-11T20:51:15.751954+0000","last_unstale":"2024-03-11T20:52:16.788338+0000","last_undegraded":"2024-03-11T20:52:16.788338+0000","last_fullsized":"2024-03-11T20:52:16.788338+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:52.008655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710578+0000","last_change":"2024-03-11T20:51:11.731064+0000","last_active":"2024-03-11T20:52:00.710578+0000","last_peered":"2024-03-11T20:52:00.710578+0000","last_clean":"2024-03-11T20:52:00.710578+0000","last_became_active":"2024-03-11T20:51:11.729651+0000","last_became_peered":"2024-03-11T20:51:11.729651+0000","last_unstale":"2024-03-11T20:52:00.710578+0000","last_undegraded":"2024-03-11T20:52:00.710578+0000","last_fullsized":"2024-03-11T20:52:00.710578+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:32:02.365119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709276+0000","last_change":"2024-03-11T20:51:13.730282+0000","last_active":"2024-03-11T20:52:00.709276+0000","last_peered":"2024-03-11T20:52:00.709276+0000","last_clean":"2024-03-11T20:52:00.709276+0000","last_became_active":"2024-03-11T20:51:13.729916+0000","last_became_peered":"2024-03-11T20:51:13.729916+0000","last_unstale":"2024-03-11T20:52:00.709276+0000","last_undegraded":"2024-03-11T20:52:00.709276+0000","last_fullsized":"2024-03-11T20:52:00.709276+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:34:26.710640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":59,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234003,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28396,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716756,"statfs":{"total":95995035648,"available":95965958144,"internally_reserved":0,"allocated":1409024,"data_stored":929902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1384448,"data_stored":928362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32004,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713148,"statfs":{"total":95995035648,"available":95962263552,"internally_reserved":0,"allocated":843776,"data_stored":339026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28396,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716756,"statfs":{"total":95995035648,"available":95965958144,"internally_reserved":0,"allocated":1413120,"data_stored":929644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":794624,"data_stored":338003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182439,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27828,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717324,"statfs":{"total":95995035648,"available":95966539776,"internally_reserved":0,"allocated":827392,"data_stored":338885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018915,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":819200,"data_stored":339074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":794624,"data_stored":338320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T20:52:29.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T20:52:29.732 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:30.199 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:29 smithi129 ceph-mon[30184]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:30.199 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:29 smithi129 ceph-mon[42066]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:30.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:29 smithi157 ceph-mon[33359]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:31.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:30 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:31.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:30 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:31.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:30 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:31.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.585+0000 7fbae7bf4700 1 -- 172.21.15.129:0/883063093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbae0105fc0 msgr2=0x7fbae010db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:31.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.585+0000 7fbae7bf4700 1 --2- 172.21.15.129:0/883063093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbae0105fc0 0x7fbae010db90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009920 tx=0x7fbad0009c30 comp rx=0 tx=0).stop 2024-03-11T20:52:31.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.585+0000 7fbae7bf4700 1 -- 172.21.15.129:0/883063093 shutdown_connections 2024-03-11T20:52:31.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.585+0000 7fbae7bf4700 1 --2- 172.21.15.129:0/883063093 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbae0105fc0 0x7fbae010db90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:31.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.585+0000 7fbae7bf4700 1 --2- 172.21.15.129:0/883063093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbae0100f70 0x7fbae0101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:31.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.585+0000 7fbae7bf4700 1 --2- 172.21.15.129:0/883063093 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbae0101920 0x7fbae01058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:31.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.585+0000 7fbae7bf4700 1 -- 172.21.15.129:0/883063093 >> 172.21.15.129:0/883063093 conn(0x7fbae0076d30 msgr2=0x7fbae0077140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:31.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.586+0000 7fbae7bf4700 1 -- 172.21.15.129:0/883063093 shutdown_connections 2024-03-11T20:52:31.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.586+0000 7fbae7bf4700 1 -- 172.21.15.129:0/883063093 wait complete. 2024-03-11T20:52:31.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.586+0000 7fbae7bf4700 1 Processor -- start 2024-03-11T20:52:31.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.586+0000 7fbae7bf4700 1 -- start start 2024-03-11T20:52:31.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae7bf4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbae0100f70 0x7fbae019c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:31.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae7bf4700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbae0101920 0x7fbae019cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:31.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae7bf4700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbae0105fc0 0x7fbae01968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:31.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae7bf4700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae0110520 con 0x7fbae0105fc0 2024-03-11T20:52:31.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae7bf4700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae01103a0 con 0x7fbae0100f70 2024-03-11T20:52:31.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae7bf4700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae01106a0 con 0x7fbae0101920 2024-03-11T20:52:31.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae518f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbae0101920 0x7fbae019cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:31.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae6191700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbae0105fc0 0x7fbae01968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:31.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae6191700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbae0105fc0 0x7fbae01968f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36190/0 (socket says 172.21.15.129:36190) 2024-03-11T20:52:31.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae518f700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbae0101920 0x7fbae019cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:35360/0 (socket says 172.21.15.129:35360) 2024-03-11T20:52:31.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae6191700 1 -- 172.21.15.129:0/690742517 learned_addr learned my addr 172.21.15.129:0/690742517 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:52:31.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae5990700 1 --2- 172.21.15.129:0/690742517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbae0100f70 0x7fbae019c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:31.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae6191700 1 -- 172.21.15.129:0/690742517 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbae0101920 msgr2=0x7fbae019cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:31.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae6191700 1 --2- 172.21.15.129:0/690742517 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbae0101920 0x7fbae019cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:31.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae6191700 1 -- 172.21.15.129:0/690742517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbae0100f70 msgr2=0x7fbae019c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:31.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae6191700 1 --2- 172.21.15.129:0/690742517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbae0100f70 0x7fbae019c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:31.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae6191700 1 -- 172.21.15.129:0/690742517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae0197050 con 0x7fbae0105fc0 2024-03-11T20:52:31.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae518f700 1 --2- 172.21.15.129:0/690742517 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbae0101920 0x7fbae019cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:52:31.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae6191700 1 --2- 172.21.15.129:0/690742517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbae0105fc0 0x7fbae01968f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009230 tx=0x7fbad000fad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:31.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbadeffd700 1 -- 172.21.15.129:0/690742517 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad000b6a0 con 0x7fbae0105fc0 2024-03-11T20:52:31.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbadeffd700 1 -- 172.21.15.129:0/690742517 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fbad0005040 con 0x7fbae0105fc0 2024-03-11T20:52:31.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbae7bf4700 1 -- 172.21.15.129:0/690742517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae01972e0 con 0x7fbae0105fc0 2024-03-11T20:52:31.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.587+0000 7fbadeffd700 1 -- 172.21.15.129:0/690742517 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad00038c0 con 0x7fbae0105fc0 2024-03-11T20:52:31.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.588+0000 7fbae7bf4700 1 -- 172.21.15.129:0/690742517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae006a320 con 0x7fbae0105fc0 2024-03-11T20:52:31.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.588+0000 7fbae7bf4700 1 -- 172.21.15.129:0/690742517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae01023c0 con 0x7fbae0105fc0 2024-03-11T20:52:31.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.589+0000 7fbadeffd700 1 -- 172.21.15.129:0/690742517 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fbad000b840 con 0x7fbae0105fc0 2024-03-11T20:52:31.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.589+0000 7fbadeffd700 1 --2- 172.21.15.129:0/690742517 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fbac806de80 0x7fbac8070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:31.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.589+0000 7fbadeffd700 1 -- 172.21.15.129:0/690742517 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7508+0+0 (secure 0 0 0) 0x7fbad0091540 con 0x7fbae0105fc0 2024-03-11T20:52:31.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.589+0000 7fbae5990700 1 --2- 172.21.15.129:0/690742517 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fbac806de80 0x7fbac8070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:31.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.590+0000 7fbae5990700 1 --2- 172.21.15.129:0/690742517 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fbac806de80 0x7fbac8070340 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbae0197a30 tx=0x7fbad800d040 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:31.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.594+0000 7fbadeffd700 1 -- 172.21.15.129:0/690742517 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fbad0013030 con 0x7fbae0105fc0 2024-03-11T20:52:31.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.805+0000 7fbae7bf4700 1 -- 172.21.15.129:0/690742517 --> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbae0197e20 con 0x7fbac806de80 2024-03-11T20:52:31.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.815+0000 7fbadeffd700 1 -- 172.21.15.129:0/690742517 <== mgr.24497 v2:172.21.15.129:6800/1191021258 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346473 (secure 0 0 0) 0x7fbae0197e20 con 0x7fbac806de80 2024-03-11T20:52:31.815 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:52:31.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.818+0000 7fbae7bf4700 1 -- 172.21.15.129:0/690742517 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fbac806de80 msgr2=0x7fbac8070340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:31.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.818+0000 7fbae7bf4700 1 --2- 172.21.15.129:0/690742517 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fbac806de80 0x7fbac8070340 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbae0197a30 tx=0x7fbad800d040 comp rx=0 tx=0).stop 2024-03-11T20:52:31.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.818+0000 7fbae7bf4700 1 -- 172.21.15.129:0/690742517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbae0105fc0 msgr2=0x7fbae01968f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:31.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.818+0000 7fbae7bf4700 1 --2- 172.21.15.129:0/690742517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbae0105fc0 0x7fbae01968f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009230 tx=0x7fbad000fad0 comp rx=0 tx=0).stop 2024-03-11T20:52:31.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.818+0000 7fbae7bf4700 1 -- 172.21.15.129:0/690742517 shutdown_connections 2024-03-11T20:52:31.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.818+0000 7fbae7bf4700 1 --2- 172.21.15.129:0/690742517 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fbac806de80 0x7fbac8070340 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:31.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.818+0000 7fbae7bf4700 1 --2- 172.21.15.129:0/690742517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbae0105fc0 0x7fbae01968f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:31.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.818+0000 7fbae7bf4700 1 --2- 172.21.15.129:0/690742517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbae0100f70 0x7fbae019c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:31.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.818+0000 7fbae7bf4700 1 --2- 172.21.15.129:0/690742517 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fbae0101920 0x7fbae019cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:31.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.818+0000 7fbae7bf4700 1 -- 172.21.15.129:0/690742517 >> 172.21.15.129:0/690742517 conn(0x7fbae0076d30 msgr2=0x7fbae00feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:31.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.819+0000 7fbae7bf4700 1 -- 172.21.15.129:0/690742517 shutdown_connections 2024-03-11T20:52:31.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:31.819+0000 7fbae7bf4700 1 -- 172.21.15.129:0/690742517 wait complete. 2024-03-11T20:52:31.819 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-03-11T20:52:32.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:31 smithi129 ceph-mon[30184]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:32.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:31 smithi129 ceph-mon[42066]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:32.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:31 smithi157 ceph-mon[33359]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:32.751 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-03-11T20:52:30.725725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1003,"num_read_kb":849,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":600,"ondisk_log_size":600,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236724,"kb_used_data":8092,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724492,"statfs":{"total":767960285184,"available":767717879808,"internally_reserved":0,"allocated":8286208,"data_stored":4481216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":234016306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002859"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707352+0000","last_change":"2024-03-11T20:51:17.761571+0000","last_active":"2024-03-11T20:52:00.707352+0000","last_peered":"2024-03-11T20:52:00.707352+0000","last_clean":"2024-03-11T20:52:00.707352+0000","last_became_active":"2024-03-11T20:51:17.761282+0000","last_became_peered":"2024-03-11T20:51:17.761282+0000","last_unstale":"2024-03-11T20:52:00.707352+0000","last_undegraded":"2024-03-11T20:52:00.707352+0000","last_fullsized":"2024-03-11T20:52:00.707352+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:17:34.237398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344609+0000","last_change":"2024-03-11T20:51:15.748116+0000","last_active":"2024-03-11T20:52:01.344609+0000","last_peered":"2024-03-11T20:52:01.344609+0000","last_clean":"2024-03-11T20:52:01.344609+0000","last_became_active":"2024-03-11T20:51:15.747469+0000","last_became_peered":"2024-03-11T20:51:15.747469+0000","last_unstale":"2024-03-11T20:52:01.344609+0000","last_undegraded":"2024-03-11T20:52:01.344609+0000","last_fullsized":"2024-03-11T20:52:01.344609+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:19:54.509317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345051+0000","last_change":"2024-03-11T20:51:11.724723+0000","last_active":"2024-03-11T20:52:01.345051+0000","last_peered":"2024-03-11T20:52:01.345051+0000","last_clean":"2024-03-11T20:52:01.345051+0000","last_became_active":"2024-03-11T20:51:11.723846+0000","last_became_peered":"2024-03-11T20:51:11.723846+0000","last_unstale":"2024-03-11T20:52:01.345051+0000","last_undegraded":"2024-03-11T20:52:01.345051+0000","last_fullsized":"2024-03-11T20:52:01.345051+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:29:31.025832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707277+0000","last_change":"2024-03-11T20:51:13.735985+0000","last_active":"2024-03-11T20:52:00.707277+0000","last_peered":"2024-03-11T20:52:00.707277+0000","last_clean":"2024-03-11T20:52:00.707277+0000","last_became_active":"2024-03-11T20:51:13.735016+0000","last_became_peered":"2024-03-11T20:51:13.735016+0000","last_unstale":"2024-03-11T20:52:00.707277+0000","last_undegraded":"2024-03-11T20:52:00.707277+0000","last_fullsized":"2024-03-11T20:52:00.707277+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:54:17.557137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710395+0000","last_change":"2024-03-11T20:51:17.891186+0000","last_active":"2024-03-11T20:52:00.710395+0000","last_peered":"2024-03-11T20:52:00.710395+0000","last_clean":"2024-03-11T20:52:00.710395+0000","last_became_active":"2024-03-11T20:51:17.890804+0000","last_became_peered":"2024-03-11T20:51:17.890804+0000","last_unstale":"2024-03-11T20:52:00.710395+0000","last_undegraded":"2024-03-11T20:52:00.710395+0000","last_fullsized":"2024-03-11T20:52:00.710395+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:52:39.048603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707279+0000","last_change":"2024-03-11T20:51:15.745388+0000","last_active":"2024-03-11T20:52:00.707279+0000","last_peered":"2024-03-11T20:52:00.707279+0000","last_clean":"2024-03-11T20:52:00.707279+0000","last_became_active":"2024-03-11T20:51:15.745100+0000","last_became_peered":"2024-03-11T20:51:15.745100+0000","last_unstale":"2024-03-11T20:52:00.707279+0000","last_undegraded":"2024-03-11T20:52:00.707279+0000","last_fullsized":"2024-03-11T20:52:00.707279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:14:47.981855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.711724+0000","last_change":"2024-03-11T20:51:11.721114+0000","last_active":"2024-03-11T20:52:00.711724+0000","last_peered":"2024-03-11T20:52:00.711724+0000","last_clean":"2024-03-11T20:52:00.711724+0000","last_became_active":"2024-03-11T20:51:11.720862+0000","last_became_peered":"2024-03-11T20:51:11.720862+0000","last_unstale":"2024-03-11T20:52:00.711724+0000","last_undegraded":"2024-03-11T20:52:00.711724+0000","last_fullsized":"2024-03-11T20:52:00.711724+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:27:34.180819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.712570+0000","last_change":"2024-03-11T20:51:13.732877+0000","last_active":"2024-03-11T20:52:00.712570+0000","last_peered":"2024-03-11T20:52:00.712570+0000","last_clean":"2024-03-11T20:52:00.712570+0000","last_became_active":"2024-03-11T20:51:13.732570+0000","last_became_peered":"2024-03-11T20:51:13.732570+0000","last_unstale":"2024-03-11T20:52:00.712570+0000","last_undegraded":"2024-03-11T20:52:00.712570+0000","last_fullsized":"2024-03-11T20:52:00.712570+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:35:01.028096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343978+0000","last_change":"2024-03-11T20:51:17.757301+0000","last_active":"2024-03-11T20:52:01.343978+0000","last_peered":"2024-03-11T20:52:01.343978+0000","last_clean":"2024-03-11T20:52:01.343978+0000","last_became_active":"2024-03-11T20:51:17.757007+0000","last_became_peered":"2024-03-11T20:51:17.757007+0000","last_unstale":"2024-03-11T20:52:01.343978+0000","last_undegraded":"2024-03-11T20:52:01.343978+0000","last_fullsized":"2024-03-11T20:52:01.343978+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:39:43.437020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821671+0000","last_change":"2024-03-11T20:51:15.750410+0000","last_active":"2024-03-11T20:52:00.821671+0000","last_peered":"2024-03-11T20:52:00.821671+0000","last_clean":"2024-03-11T20:52:00.821671+0000","last_became_active":"2024-03-11T20:51:15.749748+0000","last_became_peered":"2024-03-11T20:51:15.749748+0000","last_unstale":"2024-03-11T20:52:00.821671+0000","last_undegraded":"2024-03-11T20:52:00.821671+0000","last_fullsized":"2024-03-11T20:52:00.821671+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:35:15.056079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821911+0000","last_change":"2024-03-11T20:51:11.730768+0000","last_active":"2024-03-11T20:52:00.821911+0000","last_peered":"2024-03-11T20:52:00.821911+0000","last_clean":"2024-03-11T20:52:00.821911+0000","last_became_active":"2024-03-11T20:51:11.730460+0000","last_became_peered":"2024-03-11T20:51:11.730460+0000","last_unstale":"2024-03-11T20:52:00.821911+0000","last_undegraded":"2024-03-11T20:52:00.821911+0000","last_fullsized":"2024-03-11T20:52:00.821911+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:26:03.386745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707520+0000","last_change":"2024-03-11T20:51:13.731900+0000","last_active":"2024-03-11T20:52:00.707520+0000","last_peered":"2024-03-11T20:52:00.707520+0000","last_clean":"2024-03-11T20:52:00.707520+0000","last_became_active":"2024-03-11T20:51:13.731261+0000","last_became_peered":"2024-03-11T20:51:13.731261+0000","last_unstale":"2024-03-11T20:52:00.707520+0000","last_undegraded":"2024-03-11T20:52:00.707520+0000","last_fullsized":"2024-03-11T20:52:00.707520+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:05:37.179737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821436+0000","last_change":"2024-03-11T20:51:17.754951+0000","last_active":"2024-03-11T20:52:00.821436+0000","last_peered":"2024-03-11T20:52:00.821436+0000","last_clean":"2024-03-11T20:52:00.821436+0000","last_became_active":"2024-03-11T20:51:17.754618+0000","last_became_peered":"2024-03-11T20:51:17.754618+0000","last_unstale":"2024-03-11T20:52:00.821436+0000","last_undegraded":"2024-03-11T20:52:00.821436+0000","last_fullsized":"2024-03-11T20:52:00.821436+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:06:16.061476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828511+0000","last_change":"2024-03-11T20:51:15.744164+0000","last_active":"2024-03-11T20:52:00.828511+0000","last_peered":"2024-03-11T20:52:00.828511+0000","last_clean":"2024-03-11T20:52:00.828511+0000","last_became_active":"2024-03-11T20:51:15.743883+0000","last_became_peered":"2024-03-11T20:51:15.743883+0000","last_unstale":"2024-03-11T20:52:00.828511+0000","last_undegraded":"2024-03-11T20:52:00.828511+0000","last_fullsized":"2024-03-11T20:52:00.828511+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:26:17.521657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821819+0000","last_change":"2024-03-11T20:51:11.731273+0000","last_active":"2024-03-11T20:52:00.821819+0000","last_peered":"2024-03-11T20:52:00.821819+0000","last_clean":"2024-03-11T20:52:00.821819+0000","last_became_active":"2024-03-11T20:51:11.730970+0000","last_became_peered":"2024-03-11T20:51:11.730970+0000","last_unstale":"2024-03-11T20:52:00.821819+0000","last_undegraded":"2024-03-11T20:52:00.821819+0000","last_fullsized":"2024-03-11T20:52:00.821819+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:52:55.019056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'5","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.708794+0000","last_change":"2024-03-11T20:51:13.735898+0000","last_active":"2024-03-11T20:52:00.708794+0000","last_peered":"2024-03-11T20:52:00.708794+0000","last_clean":"2024-03-11T20:52:00.708794+0000","last_became_active":"2024-03-11T20:51:13.735549+0000","last_became_peered":"2024-03-11T20:51:13.735549+0000","last_unstale":"2024-03-11T20:52:00.708794+0000","last_undegraded":"2024-03-11T20:52:00.708794+0000","last_fullsized":"2024-03-11T20:52:00.708794+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:12:07.636129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709364+0000","last_change":"2024-03-11T20:51:17.888991+0000","last_active":"2024-03-11T20:52:00.709364+0000","last_peered":"2024-03-11T20:52:00.709364+0000","last_clean":"2024-03-11T20:52:00.709364+0000","last_became_active":"2024-03-11T20:51:17.888691+0000","last_became_peered":"2024-03-11T20:51:17.888691+0000","last_unstale":"2024-03-11T20:52:00.709364+0000","last_undegraded":"2024-03-11T20:52:00.709364+0000","last_fullsized":"2024-03-11T20:52:00.709364+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:18:23.025134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345698+0000","last_change":"2024-03-11T20:51:15.744293+0000","last_active":"2024-03-11T20:52:01.345698+0000","last_peered":"2024-03-11T20:52:01.345698+0000","last_clean":"2024-03-11T20:52:01.345698+0000","last_became_active":"2024-03-11T20:51:15.743950+0000","last_became_peered":"2024-03-11T20:51:15.743950+0000","last_unstale":"2024-03-11T20:52:01.345698+0000","last_undegraded":"2024-03-11T20:52:01.345698+0000","last_fullsized":"2024-03-11T20:52:01.345698+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:28:05.486252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344833+0000","last_change":"2024-03-11T20:51:11.730277+0000","last_active":"2024-03-11T20:52:01.344833+0000","last_peered":"2024-03-11T20:52:01.344833+0000","last_clean":"2024-03-11T20:52:01.344833+0000","last_became_active":"2024-03-11T20:51:11.729868+0000","last_became_peered":"2024-03-11T20:51:11.729868+0000","last_unstale":"2024-03-11T20:52:01.344833+0000","last_undegraded":"2024-03-11T20:52:01.344833+0000","last_fullsized":"2024-03-11T20:52:01.344833+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:28:00.331652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707532+0000","last_change":"2024-03-11T20:51:13.736663+0000","last_active":"2024-03-11T20:52:00.707532+0000","last_peered":"2024-03-11T20:52:00.707532+0000","last_clean":"2024-03-11T20:52:00.707532+0000","last_became_active":"2024-03-11T20:51:13.736316+0000","last_became_peered":"2024-03-11T20:51:13.736316+0000","last_unstale":"2024-03-11T20:52:00.707532+0000","last_undegraded":"2024-03-11T20:52:00.707532+0000","last_fullsized":"2024-03-11T20:52:00.707532+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:03:40.249599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.712433+0000","last_change":"2024-03-11T20:51:17.888988+0000","last_active":"2024-03-11T20:52:00.712433+0000","last_peered":"2024-03-11T20:52:00.712433+0000","last_clean":"2024-03-11T20:52:00.712433+0000","last_became_active":"2024-03-11T20:51:17.888587+0000","last_became_peered":"2024-03-11T20:51:17.888587+0000","last_unstale":"2024-03-11T20:52:00.712433+0000","last_undegraded":"2024-03-11T20:52:00.712433+0000","last_fullsized":"2024-03-11T20:52:00.712433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:49:49.768783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709191+0000","last_change":"2024-03-11T20:51:15.747529+0000","last_active":"2024-03-11T20:52:00.709191+0000","last_peered":"2024-03-11T20:52:00.709191+0000","last_clean":"2024-03-11T20:52:00.709191+0000","last_became_active":"2024-03-11T20:51:15.747259+0000","last_became_peered":"2024-03-11T20:51:15.747259+0000","last_unstale":"2024-03-11T20:52:00.709191+0000","last_undegraded":"2024-03-11T20:52:00.709191+0000","last_fullsized":"2024-03-11T20:52:00.709191+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:21:01.320072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707586+0000","last_change":"2024-03-11T20:51:11.727831+0000","last_active":"2024-03-11T20:52:00.707586+0000","last_peered":"2024-03-11T20:52:00.707586+0000","last_clean":"2024-03-11T20:52:00.707586+0000","last_became_active":"2024-03-11T20:51:11.727399+0000","last_became_peered":"2024-03-11T20:51:11.727399+0000","last_unstale":"2024-03-11T20:52:00.707586+0000","last_undegraded":"2024-03-11T20:52:00.707586+0000","last_fullsized":"2024-03-11T20:52:00.707586+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:54:05.940769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'12","reported_seq":52,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710984+0000","last_change":"2024-03-11T20:51:13.737828+0000","last_active":"2024-03-11T20:52:00.710984+0000","last_peered":"2024-03-11T20:52:00.710984+0000","last_clean":"2024-03-11T20:52:00.710984+0000","last_became_active":"2024-03-11T20:51:13.737487+0000","last_became_peered":"2024-03-11T20:51:13.737487+0000","last_unstale":"2024-03-11T20:52:00.710984+0000","last_undegraded":"2024-03-11T20:52:00.710984+0000","last_fullsized":"2024-03-11T20:52:00.710984+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:36:31.772982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828670+0000","last_change":"2024-03-11T20:51:17.756348+0000","last_active":"2024-03-11T20:52:00.828670+0000","last_peered":"2024-03-11T20:52:00.828670+0000","last_clean":"2024-03-11T20:52:00.828670+0000","last_became_active":"2024-03-11T20:51:17.755977+0000","last_became_peered":"2024-03-11T20:51:17.755977+0000","last_unstale":"2024-03-11T20:52:00.828670+0000","last_undegraded":"2024-03-11T20:52:00.828670+0000","last_fullsized":"2024-03-11T20:52:00.828670+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:57:52.912998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788481+0000","last_change":"2024-03-11T20:51:15.749045+0000","last_active":"2024-03-11T20:52:16.788481+0000","last_peered":"2024-03-11T20:52:16.788481+0000","last_clean":"2024-03-11T20:52:16.788481+0000","last_became_active":"2024-03-11T20:51:15.748731+0000","last_became_peered":"2024-03-11T20:51:15.748731+0000","last_unstale":"2024-03-11T20:52:16.788481+0000","last_undegraded":"2024-03-11T20:52:16.788481+0000","last_fullsized":"2024-03-11T20:52:16.788481+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:32:17.002950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344675+0000","last_change":"2024-03-11T20:51:11.729154+0000","last_active":"2024-03-11T20:52:01.344675+0000","last_peered":"2024-03-11T20:52:01.344675+0000","last_clean":"2024-03-11T20:52:01.344675+0000","last_became_active":"2024-03-11T20:51:11.728835+0000","last_became_peered":"2024-03-11T20:51:11.728835+0000","last_unstale":"2024-03-11T20:52:01.344675+0000","last_undegraded":"2024-03-11T20:52:01.344675+0000","last_fullsized":"2024-03-11T20:52:01.344675+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:52:02.829643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345323+0000","last_change":"2024-03-11T20:51:14.071941+0000","last_active":"2024-03-11T20:52:01.345323+0000","last_peered":"2024-03-11T20:52:01.345323+0000","last_clean":"2024-03-11T20:52:01.345323+0000","last_became_active":"2024-03-11T20:51:14.071283+0000","last_became_peered":"2024-03-11T20:51:14.071283+0000","last_unstale":"2024-03-11T20:52:01.345323+0000","last_undegraded":"2024-03-11T20:52:01.345323+0000","last_fullsized":"2024-03-11T20:52:01.345323+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:29:28.919667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345852+0000","last_change":"2024-03-11T20:51:17.754330+0000","last_active":"2024-03-11T20:52:01.345852+0000","last_peered":"2024-03-11T20:52:01.345852+0000","last_clean":"2024-03-11T20:52:01.345852+0000","last_became_active":"2024-03-11T20:51:17.753989+0000","last_became_peered":"2024-03-11T20:51:17.753989+0000","last_unstale":"2024-03-11T20:52:01.345852+0000","last_undegraded":"2024-03-11T20:52:01.345852+0000","last_fullsized":"2024-03-11T20:52:01.345852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:28:56.527627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345512+0000","last_change":"2024-03-11T20:51:15.748177+0000","last_active":"2024-03-11T20:52:01.345512+0000","last_peered":"2024-03-11T20:52:01.345512+0000","last_clean":"2024-03-11T20:52:01.345512+0000","last_became_active":"2024-03-11T20:51:15.747874+0000","last_became_peered":"2024-03-11T20:51:15.747874+0000","last_unstale":"2024-03-11T20:52:01.345512+0000","last_undegraded":"2024-03-11T20:52:01.345512+0000","last_fullsized":"2024-03-11T20:52:01.345512+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:07:50.572831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709418+0000","last_change":"2024-03-11T20:51:11.724549+0000","last_active":"2024-03-11T20:52:00.709418+0000","last_peered":"2024-03-11T20:52:00.709418+0000","last_clean":"2024-03-11T20:52:00.709418+0000","last_became_active":"2024-03-11T20:51:11.724267+0000","last_became_peered":"2024-03-11T20:51:11.724267+0000","last_unstale":"2024-03-11T20:52:00.709418+0000","last_undegraded":"2024-03-11T20:52:00.709418+0000","last_fullsized":"2024-03-11T20:52:00.709418+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:15:01.634207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'11","reported_seq":66,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344346+0000","last_change":"2024-03-11T20:51:31.769074+0000","last_active":"2024-03-11T20:52:01.344346+0000","last_peered":"2024-03-11T20:52:01.344346+0000","last_clean":"2024-03-11T20:52:01.344346+0000","last_became_active":"2024-03-11T20:51:31.757275+0000","last_became_peered":"2024-03-11T20:51:31.757275+0000","last_unstale":"2024-03-11T20:52:01.344346+0000","last_undegraded":"2024-03-11T20:52:01.344346+0000","last_fullsized":"2024-03-11T20:52:01.344346+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:53:29.566961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828500+0000","last_change":"2024-03-11T20:51:17.756239+0000","last_active":"2024-03-11T20:52:00.828500+0000","last_peered":"2024-03-11T20:52:00.828500+0000","last_clean":"2024-03-11T20:52:00.828500+0000","last_became_active":"2024-03-11T20:51:17.755925+0000","last_became_peered":"2024-03-11T20:51:17.755925+0000","last_unstale":"2024-03-11T20:52:00.828500+0000","last_undegraded":"2024-03-11T20:52:00.828500+0000","last_fullsized":"2024-03-11T20:52:00.828500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:35.600247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343691+0000","last_change":"2024-03-11T20:51:15.745881+0000","last_active":"2024-03-11T20:52:01.343691+0000","last_peered":"2024-03-11T20:52:01.343691+0000","last_clean":"2024-03-11T20:52:01.343691+0000","last_became_active":"2024-03-11T20:51:15.745382+0000","last_became_peered":"2024-03-11T20:51:15.745382+0000","last_unstale":"2024-03-11T20:52:01.343691+0000","last_undegraded":"2024-03-11T20:52:01.343691+0000","last_fullsized":"2024-03-11T20:52:01.343691+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:10:17.366734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344971+0000","last_change":"2024-03-11T20:51:11.724862+0000","last_active":"2024-03-11T20:52:01.344971+0000","last_peered":"2024-03-11T20:52:01.344971+0000","last_clean":"2024-03-11T20:52:01.344971+0000","last_became_active":"2024-03-11T20:51:11.724584+0000","last_became_peered":"2024-03-11T20:51:11.724584+0000","last_unstale":"2024-03-11T20:52:01.344971+0000","last_undegraded":"2024-03-11T20:52:01.344971+0000","last_fullsized":"2024-03-11T20:52:01.344971+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:38:02.538772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":35,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709522+0000","last_change":"2024-03-11T20:51:13.737953+0000","last_active":"2024-03-11T20:52:00.709522+0000","last_peered":"2024-03-11T20:52:00.709522+0000","last_clean":"2024-03-11T20:52:00.709522+0000","last_became_active":"2024-03-11T20:51:13.737476+0000","last_became_peered":"2024-03-11T20:51:13.737476+0000","last_unstale":"2024-03-11T20:52:00.709522+0000","last_undegraded":"2024-03-11T20:52:00.709522+0000","last_fullsized":"2024-03-11T20:52:00.709522+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:53:15.620913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710196+0000","last_change":"2024-03-11T20:51:17.889382+0000","last_active":"2024-03-11T20:52:00.710196+0000","last_peered":"2024-03-11T20:52:00.710196+0000","last_clean":"2024-03-11T20:52:00.710196+0000","last_became_active":"2024-03-11T20:51:17.889067+0000","last_became_peered":"2024-03-11T20:51:17.889067+0000","last_unstale":"2024-03-11T20:52:00.710196+0000","last_undegraded":"2024-03-11T20:52:00.710196+0000","last_fullsized":"2024-03-11T20:52:00.710196+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:53:45.136250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828691+0000","last_change":"2024-03-11T20:51:15.746251+0000","last_active":"2024-03-11T20:52:00.828691+0000","last_peered":"2024-03-11T20:52:00.828691+0000","last_clean":"2024-03-11T20:52:00.828691+0000","last_became_active":"2024-03-11T20:51:15.745943+0000","last_became_peered":"2024-03-11T20:51:15.745943+0000","last_unstale":"2024-03-11T20:52:00.828691+0000","last_undegraded":"2024-03-11T20:52:00.828691+0000","last_fullsized":"2024-03-11T20:52:00.828691+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:40.441153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821990+0000","last_change":"2024-03-11T20:51:11.724708+0000","last_active":"2024-03-11T20:52:00.821990+0000","last_peered":"2024-03-11T20:52:00.821990+0000","last_clean":"2024-03-11T20:52:00.821990+0000","last_became_active":"2024-03-11T20:51:11.724386+0000","last_became_peered":"2024-03-11T20:51:11.724386+0000","last_unstale":"2024-03-11T20:52:00.821990+0000","last_undegraded":"2024-03-11T20:52:00.821990+0000","last_fullsized":"2024-03-11T20:52:00.821990+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:02:43.057620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.712446+0000","last_change":"2024-03-11T20:51:14.072821+0000","last_active":"2024-03-11T20:52:00.712446+0000","last_peered":"2024-03-11T20:52:00.712446+0000","last_clean":"2024-03-11T20:52:00.712446+0000","last_became_active":"2024-03-11T20:51:14.072513+0000","last_became_peered":"2024-03-11T20:51:14.072513+0000","last_unstale":"2024-03-11T20:52:00.712446+0000","last_undegraded":"2024-03-11T20:52:00.712446+0000","last_fullsized":"2024-03-11T20:52:00.712446+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:35:06.392468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343752+0000","last_change":"2024-03-11T20:51:17.763399+0000","last_active":"2024-03-11T20:52:01.343752+0000","last_peered":"2024-03-11T20:52:01.343752+0000","last_clean":"2024-03-11T20:52:01.343752+0000","last_became_active":"2024-03-11T20:51:17.763064+0000","last_became_peered":"2024-03-11T20:51:17.763064+0000","last_unstale":"2024-03-11T20:52:01.343752+0000","last_undegraded":"2024-03-11T20:52:01.343752+0000","last_fullsized":"2024-03-11T20:52:01.343752+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:27:50.694671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710490+0000","last_change":"2024-03-11T20:51:15.747901+0000","last_active":"2024-03-11T20:52:00.710490+0000","last_peered":"2024-03-11T20:52:00.710490+0000","last_clean":"2024-03-11T20:52:00.710490+0000","last_became_active":"2024-03-11T20:51:15.747588+0000","last_became_peered":"2024-03-11T20:51:15.747588+0000","last_unstale":"2024-03-11T20:52:00.710490+0000","last_undegraded":"2024-03-11T20:52:00.710490+0000","last_fullsized":"2024-03-11T20:52:00.710490+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:46:49.002126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828774+0000","last_change":"2024-03-11T20:51:11.729623+0000","last_active":"2024-03-11T20:52:00.828774+0000","last_peered":"2024-03-11T20:52:00.828774+0000","last_clean":"2024-03-11T20:52:00.828774+0000","last_became_active":"2024-03-11T20:51:11.729216+0000","last_became_peered":"2024-03-11T20:51:11.729216+0000","last_unstale":"2024-03-11T20:52:00.828774+0000","last_undegraded":"2024-03-11T20:52:00.828774+0000","last_fullsized":"2024-03-11T20:52:00.828774+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:39:32.506304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344995+0000","last_change":"2024-03-11T20:51:13.735847+0000","last_active":"2024-03-11T20:52:01.344995+0000","last_peered":"2024-03-11T20:52:01.344995+0000","last_clean":"2024-03-11T20:52:01.344995+0000","last_became_active":"2024-03-11T20:51:13.733822+0000","last_became_peered":"2024-03-11T20:51:13.733822+0000","last_unstale":"2024-03-11T20:52:01.344995+0000","last_undegraded":"2024-03-11T20:52:01.344995+0000","last_fullsized":"2024-03-11T20:52:01.344995+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:13:19.355129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710007+0000","last_change":"2024-03-11T20:51:17.763013+0000","last_active":"2024-03-11T20:52:00.710007+0000","last_peered":"2024-03-11T20:52:00.710007+0000","last_clean":"2024-03-11T20:52:00.710007+0000","last_became_active":"2024-03-11T20:51:17.761593+0000","last_became_peered":"2024-03-11T20:51:17.761593+0000","last_unstale":"2024-03-11T20:52:00.710007+0000","last_undegraded":"2024-03-11T20:52:00.710007+0000","last_fullsized":"2024-03-11T20:52:00.710007+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:51:12.808744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344963+0000","last_change":"2024-03-11T20:51:15.749619+0000","last_active":"2024-03-11T20:52:01.344963+0000","last_peered":"2024-03-11T20:52:01.344963+0000","last_clean":"2024-03-11T20:52:01.344963+0000","last_became_active":"2024-03-11T20:51:15.749319+0000","last_became_peered":"2024-03-11T20:51:15.749319+0000","last_unstale":"2024-03-11T20:52:01.344963+0000","last_undegraded":"2024-03-11T20:52:01.344963+0000","last_fullsized":"2024-03-11T20:52:01.344963+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:51:35.586120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709472+0000","last_change":"2024-03-11T20:51:11.729146+0000","last_active":"2024-03-11T20:52:00.709472+0000","last_peered":"2024-03-11T20:52:00.709472+0000","last_clean":"2024-03-11T20:52:00.709472+0000","last_became_active":"2024-03-11T20:51:11.728793+0000","last_became_peered":"2024-03-11T20:51:11.728793+0000","last_unstale":"2024-03-11T20:52:00.709472+0000","last_undegraded":"2024-03-11T20:52:00.709472+0000","last_fullsized":"2024-03-11T20:52:00.709472+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:01:45.885696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.708977+0000","last_change":"2024-03-11T20:51:13.735203+0000","last_active":"2024-03-11T20:52:00.708977+0000","last_peered":"2024-03-11T20:52:00.708977+0000","last_clean":"2024-03-11T20:52:00.708977+0000","last_became_active":"2024-03-11T20:51:13.734811+0000","last_became_peered":"2024-03-11T20:51:13.734811+0000","last_unstale":"2024-03-11T20:52:00.708977+0000","last_undegraded":"2024-03-11T20:52:00.708977+0000","last_fullsized":"2024-03-11T20:52:00.708977+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:13:07.932358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345181+0000","last_change":"2024-03-11T20:51:17.762609+0000","last_active":"2024-03-11T20:52:01.345181+0000","last_peered":"2024-03-11T20:52:01.345181+0000","last_clean":"2024-03-11T20:52:01.345181+0000","last_became_active":"2024-03-11T20:51:17.762310+0000","last_became_peered":"2024-03-11T20:51:17.762310+0000","last_unstale":"2024-03-11T20:52:01.345181+0000","last_undegraded":"2024-03-11T20:52:01.345181+0000","last_fullsized":"2024-03-11T20:52:01.345181+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:57:03.723372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788521+0000","last_change":"2024-03-11T20:51:15.757042+0000","last_active":"2024-03-11T20:52:16.788521+0000","last_peered":"2024-03-11T20:52:16.788521+0000","last_clean":"2024-03-11T20:52:16.788521+0000","last_became_active":"2024-03-11T20:51:15.756694+0000","last_became_peered":"2024-03-11T20:51:15.756694+0000","last_unstale":"2024-03-11T20:52:16.788521+0000","last_undegraded":"2024-03-11T20:52:16.788521+0000","last_fullsized":"2024-03-11T20:52:16.788521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:22:11.775053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828655+0000","last_change":"2024-03-11T20:51:11.733581+0000","last_active":"2024-03-11T20:52:00.828655+0000","last_peered":"2024-03-11T20:52:00.828655+0000","last_clean":"2024-03-11T20:52:00.828655+0000","last_became_active":"2024-03-11T20:51:11.729712+0000","last_became_peered":"2024-03-11T20:51:11.729712+0000","last_unstale":"2024-03-11T20:52:00.828655+0000","last_undegraded":"2024-03-11T20:52:00.828655+0000","last_fullsized":"2024-03-11T20:52:00.828655+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:46:06.746085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710375+0000","last_change":"2024-03-11T20:51:14.072014+0000","last_active":"2024-03-11T20:52:00.710375+0000","last_peered":"2024-03-11T20:52:00.710375+0000","last_clean":"2024-03-11T20:52:00.710375+0000","last_became_active":"2024-03-11T20:51:14.071760+0000","last_became_peered":"2024-03-11T20:51:14.071760+0000","last_unstale":"2024-03-11T20:52:00.710375+0000","last_undegraded":"2024-03-11T20:52:00.710375+0000","last_fullsized":"2024-03-11T20:52:00.710375+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:12:17.474059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710010+0000","last_change":"2024-03-11T20:51:17.757122+0000","last_active":"2024-03-11T20:52:00.710010+0000","last_peered":"2024-03-11T20:52:00.710010+0000","last_clean":"2024-03-11T20:52:00.710010+0000","last_became_active":"2024-03-11T20:51:17.756838+0000","last_became_peered":"2024-03-11T20:51:17.756838+0000","last_unstale":"2024-03-11T20:52:00.710010+0000","last_undegraded":"2024-03-11T20:52:00.710010+0000","last_fullsized":"2024-03-11T20:52:00.710010+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:05:59.758290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788673+0000","last_change":"2024-03-11T20:51:15.748370+0000","last_active":"2024-03-11T20:52:16.788673+0000","last_peered":"2024-03-11T20:52:16.788673+0000","last_clean":"2024-03-11T20:52:16.788673+0000","last_became_active":"2024-03-11T20:51:15.747526+0000","last_became_peered":"2024-03-11T20:51:15.747526+0000","last_unstale":"2024-03-11T20:52:16.788673+0000","last_undegraded":"2024-03-11T20:52:16.788673+0000","last_fullsized":"2024-03-11T20:52:16.788673+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:30:45.786798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344312+0000","last_change":"2024-03-11T20:51:11.725253+0000","last_active":"2024-03-11T20:52:01.344312+0000","last_peered":"2024-03-11T20:52:01.344312+0000","last_clean":"2024-03-11T20:52:01.344312+0000","last_became_active":"2024-03-11T20:51:11.724851+0000","last_became_peered":"2024-03-11T20:51:11.724851+0000","last_unstale":"2024-03-11T20:52:01.344312+0000","last_undegraded":"2024-03-11T20:52:01.344312+0000","last_fullsized":"2024-03-11T20:52:01.344312+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:20:23.723816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.822140+0000","last_change":"2024-03-11T20:51:14.073927+0000","last_active":"2024-03-11T20:52:00.822140+0000","last_peered":"2024-03-11T20:52:00.822140+0000","last_clean":"2024-03-11T20:52:00.822140+0000","last_became_active":"2024-03-11T20:51:14.073517+0000","last_became_peered":"2024-03-11T20:51:14.073517+0000","last_unstale":"2024-03-11T20:52:00.822140+0000","last_undegraded":"2024-03-11T20:52:00.822140+0000","last_fullsized":"2024-03-11T20:52:00.822140+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:47:38.822484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707742+0000","last_change":"2024-03-11T20:51:17.752450+0000","last_active":"2024-03-11T20:52:00.707742+0000","last_peered":"2024-03-11T20:52:00.707742+0000","last_clean":"2024-03-11T20:52:00.707742+0000","last_became_active":"2024-03-11T20:51:17.752115+0000","last_became_peered":"2024-03-11T20:51:17.752115+0000","last_unstale":"2024-03-11T20:52:00.707742+0000","last_undegraded":"2024-03-11T20:52:00.707742+0000","last_fullsized":"2024-03-11T20:52:00.707742+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:20:26.753447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.712412+0000","last_change":"2024-03-11T20:51:15.750052+0000","last_active":"2024-03-11T20:52:00.712412+0000","last_peered":"2024-03-11T20:52:00.712412+0000","last_clean":"2024-03-11T20:52:00.712412+0000","last_became_active":"2024-03-11T20:51:15.749562+0000","last_became_peered":"2024-03-11T20:51:15.749562+0000","last_unstale":"2024-03-11T20:52:00.712412+0000","last_undegraded":"2024-03-11T20:52:00.712412+0000","last_fullsized":"2024-03-11T20:52:00.712412+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:48:54.970489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828514+0000","last_change":"2024-03-11T20:51:11.730973+0000","last_active":"2024-03-11T20:52:00.828514+0000","last_peered":"2024-03-11T20:52:00.828514+0000","last_clean":"2024-03-11T20:52:00.828514+0000","last_became_active":"2024-03-11T20:51:11.729768+0000","last_became_peered":"2024-03-11T20:51:11.729768+0000","last_unstale":"2024-03-11T20:52:00.828514+0000","last_undegraded":"2024-03-11T20:52:00.828514+0000","last_fullsized":"2024-03-11T20:52:00.828514+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:23:38.724061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345408+0000","last_change":"2024-03-11T20:51:14.072688+0000","last_active":"2024-03-11T20:52:01.345408+0000","last_peered":"2024-03-11T20:52:01.345408+0000","last_clean":"2024-03-11T20:52:01.345408+0000","last_became_active":"2024-03-11T20:51:14.072368+0000","last_became_peered":"2024-03-11T20:51:14.072368+0000","last_unstale":"2024-03-11T20:52:01.345408+0000","last_undegraded":"2024-03-11T20:52:01.345408+0000","last_fullsized":"2024-03-11T20:52:01.345408+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:43:36.363151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828920+0000","last_change":"2024-03-11T20:51:17.888736+0000","last_active":"2024-03-11T20:52:00.828920+0000","last_peered":"2024-03-11T20:52:00.828920+0000","last_clean":"2024-03-11T20:52:00.828920+0000","last_became_active":"2024-03-11T20:51:17.888438+0000","last_became_peered":"2024-03-11T20:51:17.888438+0000","last_unstale":"2024-03-11T20:52:00.828920+0000","last_undegraded":"2024-03-11T20:52:00.828920+0000","last_fullsized":"2024-03-11T20:52:00.828920+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:57:52.210447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.822014+0000","last_change":"2024-03-11T20:51:15.752669+0000","last_active":"2024-03-11T20:52:00.822014+0000","last_peered":"2024-03-11T20:52:00.822014+0000","last_clean":"2024-03-11T20:52:00.822014+0000","last_became_active":"2024-03-11T20:51:15.752393+0000","last_became_peered":"2024-03-11T20:51:15.752393+0000","last_unstale":"2024-03-11T20:52:00.822014+0000","last_undegraded":"2024-03-11T20:52:00.822014+0000","last_fullsized":"2024-03-11T20:52:00.822014+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:33:01.024179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343647+0000","last_change":"2024-03-11T20:51:11.727464+0000","last_active":"2024-03-11T20:52:01.343647+0000","last_peered":"2024-03-11T20:52:01.343647+0000","last_clean":"2024-03-11T20:52:01.343647+0000","last_became_active":"2024-03-11T20:51:11.727159+0000","last_became_peered":"2024-03-11T20:51:11.727159+0000","last_unstale":"2024-03-11T20:52:01.343647+0000","last_undegraded":"2024-03-11T20:52:01.343647+0000","last_fullsized":"2024-03-11T20:52:01.343647+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:01:05.288045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'6","reported_seq":38,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.712514+0000","last_change":"2024-03-11T20:51:14.072241+0000","last_active":"2024-03-11T20:52:00.712514+0000","last_peered":"2024-03-11T20:52:00.712514+0000","last_clean":"2024-03-11T20:52:00.712514+0000","last_became_active":"2024-03-11T20:51:14.071882+0000","last_became_peered":"2024-03-11T20:51:14.071882+0000","last_unstale":"2024-03-11T20:52:00.712514+0000","last_undegraded":"2024-03-11T20:52:00.712514+0000","last_fullsized":"2024-03-11T20:52:00.712514+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:20:56.104843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821740+0000","last_change":"2024-03-11T20:51:17.889951+0000","last_active":"2024-03-11T20:52:00.821740+0000","last_peered":"2024-03-11T20:52:00.821740+0000","last_clean":"2024-03-11T20:52:00.821740+0000","last_became_active":"2024-03-11T20:51:17.889641+0000","last_became_peered":"2024-03-11T20:51:17.889641+0000","last_unstale":"2024-03-11T20:52:00.821740+0000","last_undegraded":"2024-03-11T20:52:00.821740+0000","last_fullsized":"2024-03-11T20:52:00.821740+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:42:24.770726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'3","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.829190+0000","last_change":"2024-03-11T20:51:11.731590+0000","last_active":"2024-03-11T20:52:00.829190+0000","last_peered":"2024-03-11T20:52:00.829190+0000","last_clean":"2024-03-11T20:52:00.829190+0000","last_became_active":"2024-03-11T20:51:11.731127+0000","last_became_peered":"2024-03-11T20:51:11.731127+0000","last_unstale":"2024-03-11T20:52:00.829190+0000","last_undegraded":"2024-03-11T20:52:00.829190+0000","last_fullsized":"2024-03-11T20:52:00.829190+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:40:10.498058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1003,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821452+0000","last_change":"2024-03-11T20:51:14.075015+0000","last_active":"2024-03-11T20:52:00.821452+0000","last_peered":"2024-03-11T20:52:00.821452+0000","last_clean":"2024-03-11T20:52:00.821452+0000","last_became_active":"2024-03-11T20:51:14.074664+0000","last_became_peered":"2024-03-11T20:51:14.074664+0000","last_unstale":"2024-03-11T20:52:00.821452+0000","last_undegraded":"2024-03-11T20:52:00.821452+0000","last_fullsized":"2024-03-11T20:52:00.821452+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:05.021441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":51,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788495+0000","last_change":"2024-03-11T20:51:15.751443+0000","last_active":"2024-03-11T20:52:16.788495+0000","last_peered":"2024-03-11T20:52:16.788495+0000","last_clean":"2024-03-11T20:52:16.788495+0000","last_became_active":"2024-03-11T20:51:15.751142+0000","last_became_peered":"2024-03-11T20:51:15.751142+0000","last_unstale":"2024-03-11T20:52:16.788495+0000","last_undegraded":"2024-03-11T20:52:16.788495+0000","last_fullsized":"2024-03-11T20:52:16.788495+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:09:44.328862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710302+0000","last_change":"2024-03-11T20:51:17.757771+0000","last_active":"2024-03-11T20:52:00.710302+0000","last_peered":"2024-03-11T20:52:00.710302+0000","last_clean":"2024-03-11T20:52:00.710302+0000","last_became_active":"2024-03-11T20:51:17.757456+0000","last_became_peered":"2024-03-11T20:51:17.757456+0000","last_unstale":"2024-03-11T20:52:00.710302+0000","last_undegraded":"2024-03-11T20:52:00.710302+0000","last_fullsized":"2024-03-11T20:52:00.710302+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:24:17.674062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.829118+0000","last_change":"2024-03-11T20:51:11.730704+0000","last_active":"2024-03-11T20:52:00.829118+0000","last_peered":"2024-03-11T20:52:00.829118+0000","last_clean":"2024-03-11T20:52:00.829118+0000","last_became_active":"2024-03-11T20:51:11.729468+0000","last_became_peered":"2024-03-11T20:51:11.729468+0000","last_unstale":"2024-03-11T20:52:00.829118+0000","last_undegraded":"2024-03-11T20:52:00.829118+0000","last_fullsized":"2024-03-11T20:52:00.829118+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:00:58.165332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'12","reported_seq":70,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707770+0000","last_change":"2024-03-11T20:51:31.762808+0000","last_active":"2024-03-11T20:52:00.707770+0000","last_peered":"2024-03-11T20:52:00.707770+0000","last_clean":"2024-03-11T20:52:00.707770+0000","last_became_active":"2024-03-11T20:51:31.756369+0000","last_became_peered":"2024-03-11T20:51:31.756369+0000","last_unstale":"2024-03-11T20:52:00.707770+0000","last_undegraded":"2024-03-11T20:52:00.707770+0000","last_fullsized":"2024-03-11T20:52:00.707770+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:14:01.083562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.706815+0000","last_change":"2024-03-11T20:51:15.745193+0000","last_active":"2024-03-11T20:52:00.706815+0000","last_peered":"2024-03-11T20:52:00.706815+0000","last_clean":"2024-03-11T20:52:00.706815+0000","last_became_active":"2024-03-11T20:51:15.744733+0000","last_became_peered":"2024-03-11T20:51:15.744733+0000","last_unstale":"2024-03-11T20:52:00.706815+0000","last_undegraded":"2024-03-11T20:52:00.706815+0000","last_fullsized":"2024-03-11T20:52:00.706815+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:03:33.739180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828341+0000","last_change":"2024-03-11T20:51:17.759356+0000","last_active":"2024-03-11T20:52:00.828341+0000","last_peered":"2024-03-11T20:52:00.828341+0000","last_clean":"2024-03-11T20:52:00.828341+0000","last_became_active":"2024-03-11T20:51:17.759097+0000","last_became_peered":"2024-03-11T20:51:17.759097+0000","last_unstale":"2024-03-11T20:52:00.828341+0000","last_undegraded":"2024-03-11T20:52:00.828341+0000","last_fullsized":"2024-03-11T20:52:00.828341+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:50:15.438548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707495+0000","last_change":"2024-03-11T20:51:15.750672+0000","last_active":"2024-03-11T20:52:00.707495+0000","last_peered":"2024-03-11T20:52:00.707495+0000","last_clean":"2024-03-11T20:52:00.707495+0000","last_became_active":"2024-03-11T20:51:15.750295+0000","last_became_peered":"2024-03-11T20:51:15.750295+0000","last_unstale":"2024-03-11T20:52:00.707495+0000","last_undegraded":"2024-03-11T20:52:00.707495+0000","last_fullsized":"2024-03-11T20:52:00.707495+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:42:38.604383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.829039+0000","last_change":"2024-03-11T20:51:11.727910+0000","last_active":"2024-03-11T20:52:00.829039+0000","last_peered":"2024-03-11T20:52:00.829039+0000","last_clean":"2024-03-11T20:52:00.829039+0000","last_became_active":"2024-03-11T20:51:11.727576+0000","last_became_peered":"2024-03-11T20:51:11.727576+0000","last_unstale":"2024-03-11T20:52:00.829039+0000","last_undegraded":"2024-03-11T20:52:00.829039+0000","last_fullsized":"2024-03-11T20:52:00.829039+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:41.460100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'19","reported_seq":60,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343837+0000","last_change":"2024-03-11T20:51:14.073803+0000","last_active":"2024-03-11T20:52:01.343837+0000","last_peered":"2024-03-11T20:52:01.343837+0000","last_clean":"2024-03-11T20:52:01.343837+0000","last_became_active":"2024-03-11T20:51:14.073462+0000","last_became_peered":"2024-03-11T20:51:14.073462+0000","last_unstale":"2024-03-11T20:52:01.343837+0000","last_undegraded":"2024-03-11T20:52:01.343837+0000","last_fullsized":"2024-03-11T20:52:01.343837+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:27:19.898543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343993+0000","last_change":"2024-03-11T20:51:17.760664+0000","last_active":"2024-03-11T20:52:01.343993+0000","last_peered":"2024-03-11T20:52:01.343993+0000","last_clean":"2024-03-11T20:52:01.343993+0000","last_became_active":"2024-03-11T20:51:17.760350+0000","last_became_peered":"2024-03-11T20:51:17.760350+0000","last_unstale":"2024-03-11T20:52:01.343993+0000","last_undegraded":"2024-03-11T20:52:01.343993+0000","last_fullsized":"2024-03-11T20:52:01.343993+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:34:28.076159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707280+0000","last_change":"2024-03-11T20:51:15.746397+0000","last_active":"2024-03-11T20:52:00.707280+0000","last_peered":"2024-03-11T20:52:00.707280+0000","last_clean":"2024-03-11T20:52:00.707280+0000","last_became_active":"2024-03-11T20:51:15.746074+0000","last_became_peered":"2024-03-11T20:51:15.746074+0000","last_unstale":"2024-03-11T20:52:00.707280+0000","last_undegraded":"2024-03-11T20:52:00.707280+0000","last_fullsized":"2024-03-11T20:52:00.707280+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:07:12.110877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821823+0000","last_change":"2024-03-11T20:51:11.726209+0000","last_active":"2024-03-11T20:52:00.821823+0000","last_peered":"2024-03-11T20:52:00.821823+0000","last_clean":"2024-03-11T20:52:00.821823+0000","last_became_active":"2024-03-11T20:51:11.725872+0000","last_became_peered":"2024-03-11T20:51:11.725872+0000","last_unstale":"2024-03-11T20:52:00.821823+0000","last_undegraded":"2024-03-11T20:52:00.821823+0000","last_fullsized":"2024-03-11T20:52:00.821823+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:34:31.051009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'9","reported_seq":45,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344661+0000","last_change":"2024-03-11T20:51:13.735310+0000","last_active":"2024-03-11T20:52:01.344661+0000","last_peered":"2024-03-11T20:52:01.344661+0000","last_clean":"2024-03-11T20:52:01.344661+0000","last_became_active":"2024-03-11T20:51:13.734663+0000","last_became_peered":"2024-03-11T20:51:13.734663+0000","last_unstale":"2024-03-11T20:52:01.344661+0000","last_undegraded":"2024-03-11T20:52:01.344661+0000","last_fullsized":"2024-03-11T20:52:01.344661+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:01:57.099493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710874+0000","last_change":"2024-03-11T20:51:17.756929+0000","last_active":"2024-03-11T20:52:00.710874+0000","last_peered":"2024-03-11T20:52:00.710874+0000","last_clean":"2024-03-11T20:52:00.710874+0000","last_became_active":"2024-03-11T20:51:17.756629+0000","last_became_peered":"2024-03-11T20:51:17.756629+0000","last_unstale":"2024-03-11T20:52:00.710874+0000","last_undegraded":"2024-03-11T20:52:00.710874+0000","last_fullsized":"2024-03-11T20:52:00.710874+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:49:01.858696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821578+0000","last_change":"2024-03-11T20:51:14.733852+0000","last_active":"2024-03-11T20:52:00.821578+0000","last_peered":"2024-03-11T20:52:00.821578+0000","last_clean":"2024-03-11T20:52:00.821578+0000","last_became_active":"2024-03-11T20:51:10.707493+0000","last_became_peered":"2024-03-11T20:51:10.707493+0000","last_unstale":"2024-03-11T20:52:00.821578+0000","last_undegraded":"2024-03-11T20:52:00.821578+0000","last_fullsized":"2024-03-11T20:52:00.821578+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:09.689152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:09.689152+0000","last_clean_scrub_stamp":"2024-03-11T20:51:09.689152+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:06:20.986387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001974848,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.708598+0000","last_change":"2024-03-11T20:51:11.728606+0000","last_active":"2024-03-11T20:52:00.708598+0000","last_peered":"2024-03-11T20:52:00.708598+0000","last_clean":"2024-03-11T20:52:00.708598+0000","last_became_active":"2024-03-11T20:51:11.728289+0000","last_became_peered":"2024-03-11T20:51:11.728289+0000","last_unstale":"2024-03-11T20:52:00.708598+0000","last_undegraded":"2024-03-11T20:52:00.708598+0000","last_fullsized":"2024-03-11T20:52:00.708598+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:30:44.896019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"68'28","reported_seq":92,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788341+0000","last_change":"2024-03-11T20:51:13.739087+0000","last_active":"2024-03-11T20:52:16.788341+0000","last_peered":"2024-03-11T20:52:16.788341+0000","last_clean":"2024-03-11T20:52:16.788341+0000","last_became_active":"2024-03-11T20:51:13.738746+0000","last_became_peered":"2024-03-11T20:51:13.738746+0000","last_unstale":"2024-03-11T20:52:16.788341+0000","last_undegraded":"2024-03-11T20:52:16.788341+0000","last_fullsized":"2024-03-11T20:52:16.788341+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:57:23.938142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344600+0000","last_change":"2024-03-11T20:51:15.747401+0000","last_active":"2024-03-11T20:52:01.344600+0000","last_peered":"2024-03-11T20:52:01.344600+0000","last_clean":"2024-03-11T20:52:01.344600+0000","last_became_active":"2024-03-11T20:51:15.747010+0000","last_became_peered":"2024-03-11T20:51:15.747010+0000","last_unstale":"2024-03-11T20:52:01.344600+0000","last_undegraded":"2024-03-11T20:52:01.344600+0000","last_fullsized":"2024-03-11T20:52:01.344600+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:42:50.389808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.829096+0000","last_change":"2024-03-11T20:51:17.890420+0000","last_active":"2024-03-11T20:52:00.829096+0000","last_peered":"2024-03-11T20:52:00.829096+0000","last_clean":"2024-03-11T20:52:00.829096+0000","last_became_active":"2024-03-11T20:51:17.889919+0000","last_became_peered":"2024-03-11T20:51:17.889919+0000","last_unstale":"2024-03-11T20:52:00.829096+0000","last_undegraded":"2024-03-11T20:52:00.829096+0000","last_fullsized":"2024-03-11T20:52:00.829096+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:55:01.662146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344764+0000","last_change":"2024-03-11T20:51:11.723344+0000","last_active":"2024-03-11T20:52:01.344764+0000","last_peered":"2024-03-11T20:52:01.344764+0000","last_clean":"2024-03-11T20:52:01.344764+0000","last_became_active":"2024-03-11T20:51:11.722921+0000","last_became_peered":"2024-03-11T20:51:11.722921+0000","last_unstale":"2024-03-11T20:52:01.344764+0000","last_undegraded":"2024-03-11T20:52:01.344764+0000","last_fullsized":"2024-03-11T20:52:01.344764+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:30:48.721567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707070+0000","last_change":"2024-03-11T20:51:14.728335+0000","last_active":"2024-03-11T20:52:00.707070+0000","last_peered":"2024-03-11T20:52:00.707070+0000","last_clean":"2024-03-11T20:52:00.707070+0000","last_became_active":"2024-03-11T20:51:10.709107+0000","last_became_peered":"2024-03-11T20:51:10.709107+0000","last_unstale":"2024-03-11T20:52:00.707070+0000","last_undegraded":"2024-03-11T20:52:00.707070+0000","last_fullsized":"2024-03-11T20:52:00.707070+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:09.689152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:09.689152+0000","last_clean_scrub_stamp":"2024-03-11T20:51:09.689152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:53:21.681104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054735999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"68'13","reported_seq":56,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345761+0000","last_change":"2024-03-11T20:51:13.736954+0000","last_active":"2024-03-11T20:52:01.345761+0000","last_peered":"2024-03-11T20:52:01.345761+0000","last_clean":"2024-03-11T20:52:01.345761+0000","last_became_active":"2024-03-11T20:51:13.736669+0000","last_became_peered":"2024-03-11T20:51:13.736669+0000","last_unstale":"2024-03-11T20:52:01.345761+0000","last_undegraded":"2024-03-11T20:52:01.345761+0000","last_fullsized":"2024-03-11T20:52:01.345761+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:55:46.337370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.708120+0000","last_change":"2024-03-11T20:51:15.751380+0000","last_active":"2024-03-11T20:52:00.708120+0000","last_peered":"2024-03-11T20:52:00.708120+0000","last_clean":"2024-03-11T20:52:00.708120+0000","last_became_active":"2024-03-11T20:51:15.751088+0000","last_became_peered":"2024-03-11T20:51:15.751088+0000","last_unstale":"2024-03-11T20:52:00.708120+0000","last_undegraded":"2024-03-11T20:52:00.708120+0000","last_fullsized":"2024-03-11T20:52:00.708120+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:35:14.185900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.822112+0000","last_change":"2024-03-11T20:51:17.754996+0000","last_active":"2024-03-11T20:52:00.822112+0000","last_peered":"2024-03-11T20:52:00.822112+0000","last_clean":"2024-03-11T20:52:00.822112+0000","last_became_active":"2024-03-11T20:51:17.754690+0000","last_became_peered":"2024-03-11T20:51:17.754690+0000","last_unstale":"2024-03-11T20:52:00.822112+0000","last_undegraded":"2024-03-11T20:52:00.822112+0000","last_fullsized":"2024-03-11T20:52:00.822112+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:48:06.180451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"72'128","reported_seq":278,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:02.806506+0000","last_change":"2024-03-11T20:51:01.493203+0000","last_active":"2024-03-11T20:52:02.806506+0000","last_peered":"2024-03-11T20:52:02.806506+0000","last_clean":"2024-03-11T20:52:02.806506+0000","last_became_active":"2024-03-11T20:51:01.492723+0000","last_became_peered":"2024-03-11T20:51:01.492723+0000","last_unstale":"2024-03-11T20:52:02.806506+0000","last_undegraded":"2024-03-11T20:52:02.806506+0000","last_fullsized":"2024-03-11T20:52:02.806506+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:48:50.017181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:48:50.017181+0000","last_clean_scrub_stamp":"2024-03-11T20:48:50.017181+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:21:38.769263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"61'1","reported_seq":39,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.711847+0000","last_change":"2024-03-11T20:51:11.723644+0000","last_active":"2024-03-11T20:52:00.711847+0000","last_peered":"2024-03-11T20:52:00.711847+0000","last_clean":"2024-03-11T20:52:00.711847+0000","last_became_active":"2024-03-11T20:51:11.723310+0000","last_became_peered":"2024-03-11T20:51:11.723310+0000","last_unstale":"2024-03-11T20:52:00.711847+0000","last_undegraded":"2024-03-11T20:52:00.711847+0000","last_fullsized":"2024-03-11T20:52:00.711847+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:11:33.944833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":66,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788597+0000","last_change":"2024-03-11T20:51:14.072215+0000","last_active":"2024-03-11T20:52:16.788597+0000","last_peered":"2024-03-11T20:52:16.788597+0000","last_clean":"2024-03-11T20:52:16.788597+0000","last_became_active":"2024-03-11T20:51:14.071826+0000","last_became_peered":"2024-03-11T20:51:14.071826+0000","last_unstale":"2024-03-11T20:52:16.788597+0000","last_undegraded":"2024-03-11T20:52:16.788597+0000","last_fullsized":"2024-03-11T20:52:16.788597+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:31:42.729179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828782+0000","last_change":"2024-03-11T20:51:15.748126+0000","last_active":"2024-03-11T20:52:00.828782+0000","last_peered":"2024-03-11T20:52:00.828782+0000","last_clean":"2024-03-11T20:52:00.828782+0000","last_became_active":"2024-03-11T20:51:15.747840+0000","last_became_peered":"2024-03-11T20:51:15.747840+0000","last_unstale":"2024-03-11T20:52:00.828782+0000","last_undegraded":"2024-03-11T20:52:00.828782+0000","last_fullsized":"2024-03-11T20:52:00.828782+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:14:29.960588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345140+0000","last_change":"2024-03-11T20:51:17.757756+0000","last_active":"2024-03-11T20:52:01.345140+0000","last_peered":"2024-03-11T20:52:01.345140+0000","last_clean":"2024-03-11T20:52:01.345140+0000","last_became_active":"2024-03-11T20:51:17.757474+0000","last_became_peered":"2024-03-11T20:51:17.757474+0000","last_unstale":"2024-03-11T20:52:01.345140+0000","last_undegraded":"2024-03-11T20:52:01.345140+0000","last_fullsized":"2024-03-11T20:52:01.345140+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:50:07.033497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344214+0000","last_change":"2024-03-11T20:51:11.729185+0000","last_active":"2024-03-11T20:52:01.344214+0000","last_peered":"2024-03-11T20:52:01.344214+0000","last_clean":"2024-03-11T20:52:01.344214+0000","last_became_active":"2024-03-11T20:51:11.728866+0000","last_became_peered":"2024-03-11T20:51:11.728866+0000","last_unstale":"2024-03-11T20:52:01.344214+0000","last_undegraded":"2024-03-11T20:52:01.344214+0000","last_fullsized":"2024-03-11T20:52:01.344214+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:38:44.383310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"69'5","reported_seq":114,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:25.237942+0000","last_change":"2024-03-11T20:51:14.734085+0000","last_active":"2024-03-11T20:52:25.237942+0000","last_peered":"2024-03-11T20:52:25.237942+0000","last_clean":"2024-03-11T20:52:25.237942+0000","last_became_active":"2024-03-11T20:51:10.707227+0000","last_became_peered":"2024-03-11T20:51:10.707227+0000","last_unstale":"2024-03-11T20:52:25.237942+0000","last_undegraded":"2024-03-11T20:52:25.237942+0000","last_fullsized":"2024-03-11T20:52:25.237942+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:09.689152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:09.689152+0000","last_clean_scrub_stamp":"2024-03-11T20:51:09.689152+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:05:07.734778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046710399999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":64,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":47,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345181+0000","last_change":"2024-03-11T20:51:13.735009+0000","last_active":"2024-03-11T20:52:01.345181+0000","last_peered":"2024-03-11T20:52:01.345181+0000","last_clean":"2024-03-11T20:52:01.345181+0000","last_became_active":"2024-03-11T20:51:13.734654+0000","last_became_peered":"2024-03-11T20:51:13.734654+0000","last_unstale":"2024-03-11T20:52:01.345181+0000","last_undegraded":"2024-03-11T20:52:01.345181+0000","last_fullsized":"2024-03-11T20:52:01.345181+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T20:55:49.732038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.822235+0000","last_change":"2024-03-11T20:51:15.756612+0000","last_active":"2024-03-11T20:52:00.822235+0000","last_peered":"2024-03-11T20:52:00.822235+0000","last_clean":"2024-03-11T20:52:00.822235+0000","last_became_active":"2024-03-11T20:51:15.756110+0000","last_became_peered":"2024-03-11T20:51:15.756110+0000","last_unstale":"2024-03-11T20:52:00.822235+0000","last_undegraded":"2024-03-11T20:52:00.822235+0000","last_fullsized":"2024-03-11T20:52:00.822235+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:24:54.185806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344572+0000","last_change":"2024-03-11T20:51:17.888814+0000","last_active":"2024-03-11T20:52:01.344572+0000","last_peered":"2024-03-11T20:52:01.344572+0000","last_clean":"2024-03-11T20:52:01.344572+0000","last_became_active":"2024-03-11T20:51:17.888460+0000","last_became_peered":"2024-03-11T20:51:17.888460+0000","last_unstale":"2024-03-11T20:52:01.344572+0000","last_undegraded":"2024-03-11T20:52:01.344572+0000","last_fullsized":"2024-03-11T20:52:01.344572+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:59:29.851934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345345+0000","last_change":"2024-03-11T20:51:11.714853+0000","last_active":"2024-03-11T20:52:01.345345+0000","last_peered":"2024-03-11T20:52:01.345345+0000","last_clean":"2024-03-11T20:52:01.345345+0000","last_became_active":"2024-03-11T20:51:11.714501+0000","last_became_peered":"2024-03-11T20:51:11.714501+0000","last_unstale":"2024-03-11T20:52:01.345345+0000","last_undegraded":"2024-03-11T20:52:01.345345+0000","last_fullsized":"2024-03-11T20:52:01.345345+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:37.956357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'19","reported_seq":65,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.344919+0000","last_change":"2024-03-11T20:51:14.071827+0000","last_active":"2024-03-11T20:52:01.344919+0000","last_peered":"2024-03-11T20:52:01.344919+0000","last_clean":"2024-03-11T20:52:01.344919+0000","last_became_active":"2024-03-11T20:51:14.071470+0000","last_became_peered":"2024-03-11T20:51:14.071470+0000","last_unstale":"2024-03-11T20:52:01.344919+0000","last_undegraded":"2024-03-11T20:52:01.344919+0000","last_fullsized":"2024-03-11T20:52:01.344919+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:03:02.382794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343923+0000","last_change":"2024-03-11T20:51:15.751107+0000","last_active":"2024-03-11T20:52:01.343923+0000","last_peered":"2024-03-11T20:52:01.343923+0000","last_clean":"2024-03-11T20:52:01.343923+0000","last_became_active":"2024-03-11T20:51:15.750808+0000","last_became_peered":"2024-03-11T20:51:15.750808+0000","last_unstale":"2024-03-11T20:52:01.343923+0000","last_undegraded":"2024-03-11T20:52:01.343923+0000","last_fullsized":"2024-03-11T20:52:01.343923+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:55:22.651002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343760+0000","last_change":"2024-03-11T20:51:17.762486+0000","last_active":"2024-03-11T20:52:01.343760+0000","last_peered":"2024-03-11T20:52:01.343760+0000","last_clean":"2024-03-11T20:52:01.343760+0000","last_became_active":"2024-03-11T20:51:17.762153+0000","last_became_peered":"2024-03-11T20:51:17.762153+0000","last_unstale":"2024-03-11T20:52:01.343760+0000","last_undegraded":"2024-03-11T20:52:01.343760+0000","last_fullsized":"2024-03-11T20:52:01.343760+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:57:38.439624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.822018+0000","last_change":"2024-03-11T20:51:11.723320+0000","last_active":"2024-03-11T20:52:00.822018+0000","last_peered":"2024-03-11T20:52:00.822018+0000","last_clean":"2024-03-11T20:52:00.822018+0000","last_became_active":"2024-03-11T20:51:11.722992+0000","last_became_peered":"2024-03-11T20:51:11.722992+0000","last_unstale":"2024-03-11T20:52:00.822018+0000","last_undegraded":"2024-03-11T20:52:00.822018+0000","last_fullsized":"2024-03-11T20:52:00.822018+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:16.876429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'10","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.706972+0000","last_change":"2024-03-11T20:51:31.763694+0000","last_active":"2024-03-11T20:52:00.706972+0000","last_peered":"2024-03-11T20:52:00.706972+0000","last_clean":"2024-03-11T20:52:00.706972+0000","last_became_active":"2024-03-11T20:51:31.756525+0000","last_became_peered":"2024-03-11T20:51:31.756525+0000","last_unstale":"2024-03-11T20:52:00.706972+0000","last_undegraded":"2024-03-11T20:52:00.706972+0000","last_fullsized":"2024-03-11T20:52:00.706972+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:09:45.138476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788331+0000","last_change":"2024-03-11T20:51:15.753138+0000","last_active":"2024-03-11T20:52:16.788331+0000","last_peered":"2024-03-11T20:52:16.788331+0000","last_clean":"2024-03-11T20:52:16.788331+0000","last_became_active":"2024-03-11T20:51:15.752789+0000","last_became_peered":"2024-03-11T20:51:15.752789+0000","last_unstale":"2024-03-11T20:52:16.788331+0000","last_undegraded":"2024-03-11T20:52:16.788331+0000","last_fullsized":"2024-03-11T20:52:16.788331+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:49:48.857603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.828889+0000","last_change":"2024-03-11T20:51:17.890460+0000","last_active":"2024-03-11T20:52:00.828889+0000","last_peered":"2024-03-11T20:52:00.828889+0000","last_clean":"2024-03-11T20:52:00.828889+0000","last_became_active":"2024-03-11T20:51:17.890176+0000","last_became_peered":"2024-03-11T20:51:17.890176+0000","last_unstale":"2024-03-11T20:52:00.828889+0000","last_undegraded":"2024-03-11T20:52:00.828889+0000","last_fullsized":"2024-03-11T20:52:00.828889+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:14:14.090671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":34,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345293+0000","last_change":"2024-03-11T20:51:11.729566+0000","last_active":"2024-03-11T20:52:01.345293+0000","last_peered":"2024-03-11T20:52:01.345293+0000","last_clean":"2024-03-11T20:52:01.345293+0000","last_became_active":"2024-03-11T20:51:11.728869+0000","last_became_peered":"2024-03-11T20:51:11.728869+0000","last_unstale":"2024-03-11T20:52:01.345293+0000","last_undegraded":"2024-03-11T20:52:01.345293+0000","last_fullsized":"2024-03-11T20:52:01.345293+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:49:57.454894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"68'14","reported_seq":50,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707567+0000","last_change":"2024-03-11T20:51:13.730891+0000","last_active":"2024-03-11T20:52:00.707567+0000","last_peered":"2024-03-11T20:52:00.707567+0000","last_clean":"2024-03-11T20:52:00.707567+0000","last_became_active":"2024-03-11T20:51:13.730578+0000","last_became_peered":"2024-03-11T20:51:13.730578+0000","last_unstale":"2024-03-11T20:52:00.707567+0000","last_undegraded":"2024-03-11T20:52:00.707567+0000","last_fullsized":"2024-03-11T20:52:00.707567+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:57:59.848193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788329+0000","last_change":"2024-03-11T20:51:15.751998+0000","last_active":"2024-03-11T20:52:16.788329+0000","last_peered":"2024-03-11T20:52:16.788329+0000","last_clean":"2024-03-11T20:52:16.788329+0000","last_became_active":"2024-03-11T20:51:15.751637+0000","last_became_peered":"2024-03-11T20:51:15.751637+0000","last_unstale":"2024-03-11T20:52:16.788329+0000","last_undegraded":"2024-03-11T20:52:16.788329+0000","last_fullsized":"2024-03-11T20:52:16.788329+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:39:03.303229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707490+0000","last_change":"2024-03-11T20:51:17.757133+0000","last_active":"2024-03-11T20:52:00.707490+0000","last_peered":"2024-03-11T20:52:00.707490+0000","last_clean":"2024-03-11T20:52:00.707490+0000","last_became_active":"2024-03-11T20:51:17.756830+0000","last_became_peered":"2024-03-11T20:51:17.756830+0000","last_unstale":"2024-03-11T20:52:00.707490+0000","last_undegraded":"2024-03-11T20:52:00.707490+0000","last_fullsized":"2024-03-11T20:52:00.707490+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:32:16.808215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.711380+0000","last_change":"2024-03-11T20:51:11.725766+0000","last_active":"2024-03-11T20:52:00.711380+0000","last_peered":"2024-03-11T20:52:00.711380+0000","last_clean":"2024-03-11T20:52:00.711380+0000","last_became_active":"2024-03-11T20:51:11.725376+0000","last_became_peered":"2024-03-11T20:51:11.725376+0000","last_unstale":"2024-03-11T20:52:00.711380+0000","last_undegraded":"2024-03-11T20:52:00.711380+0000","last_fullsized":"2024-03-11T20:52:00.711380+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:29:50.920405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'18","reported_seq":61,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.711298+0000","last_change":"2024-03-11T20:51:14.072597+0000","last_active":"2024-03-11T20:52:00.711298+0000","last_peered":"2024-03-11T20:52:00.711298+0000","last_clean":"2024-03-11T20:52:00.711298+0000","last_became_active":"2024-03-11T20:51:14.072354+0000","last_became_peered":"2024-03-11T20:51:14.072354+0000","last_unstale":"2024-03-11T20:52:00.711298+0000","last_undegraded":"2024-03-11T20:52:00.711298+0000","last_fullsized":"2024-03-11T20:52:00.711298+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:59:53.066834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707326+0000","last_change":"2024-03-11T20:51:15.745710+0000","last_active":"2024-03-11T20:52:00.707326+0000","last_peered":"2024-03-11T20:52:00.707326+0000","last_clean":"2024-03-11T20:52:00.707326+0000","last_became_active":"2024-03-11T20:51:15.745425+0000","last_became_peered":"2024-03-11T20:51:15.745425+0000","last_unstale":"2024-03-11T20:52:00.707326+0000","last_undegraded":"2024-03-11T20:52:00.707326+0000","last_fullsized":"2024-03-11T20:52:00.707326+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:13:41.155899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.821825+0000","last_change":"2024-03-11T20:51:17.758624+0000","last_active":"2024-03-11T20:52:00.821825+0000","last_peered":"2024-03-11T20:52:00.821825+0000","last_clean":"2024-03-11T20:52:00.821825+0000","last_became_active":"2024-03-11T20:51:17.758205+0000","last_became_peered":"2024-03-11T20:51:17.758205+0000","last_unstale":"2024-03-11T20:52:00.821825+0000","last_undegraded":"2024-03-11T20:52:00.821825+0000","last_fullsized":"2024-03-11T20:52:00.821825+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:30:21.138507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788501+0000","last_change":"2024-03-11T20:51:15.752710+0000","last_active":"2024-03-11T20:52:16.788501+0000","last_peered":"2024-03-11T20:52:16.788501+0000","last_clean":"2024-03-11T20:52:16.788501+0000","last_became_active":"2024-03-11T20:51:15.752429+0000","last_became_peered":"2024-03-11T20:51:15.752429+0000","last_unstale":"2024-03-11T20:52:16.788501+0000","last_undegraded":"2024-03-11T20:52:16.788501+0000","last_fullsized":"2024-03-11T20:52:16.788501+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:41:42.487209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709631+0000","last_change":"2024-03-11T20:51:11.722958+0000","last_active":"2024-03-11T20:52:00.709631+0000","last_peered":"2024-03-11T20:52:00.709631+0000","last_clean":"2024-03-11T20:52:00.709631+0000","last_became_active":"2024-03-11T20:51:11.722655+0000","last_became_peered":"2024-03-11T20:51:11.722655+0000","last_unstale":"2024-03-11T20:52:00.709631+0000","last_undegraded":"2024-03-11T20:52:00.709631+0000","last_fullsized":"2024-03-11T20:52:00.709631+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:16:42.699212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":54,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345943+0000","last_change":"2024-03-11T20:51:13.738088+0000","last_active":"2024-03-11T20:52:01.345943+0000","last_peered":"2024-03-11T20:52:01.345943+0000","last_clean":"2024-03-11T20:52:01.345943+0000","last_became_active":"2024-03-11T20:51:13.737794+0000","last_became_peered":"2024-03-11T20:51:13.737794+0000","last_unstale":"2024-03-11T20:52:01.345943+0000","last_undegraded":"2024-03-11T20:52:01.345943+0000","last_fullsized":"2024-03-11T20:52:01.345943+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:58:23.606555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710278+0000","last_change":"2024-03-11T20:51:17.888681+0000","last_active":"2024-03-11T20:52:00.710278+0000","last_peered":"2024-03-11T20:52:00.710278+0000","last_clean":"2024-03-11T20:52:00.710278+0000","last_became_active":"2024-03-11T20:51:17.888347+0000","last_became_peered":"2024-03-11T20:51:17.888347+0000","last_unstale":"2024-03-11T20:52:00.710278+0000","last_undegraded":"2024-03-11T20:52:00.710278+0000","last_fullsized":"2024-03-11T20:52:00.710278+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:06:28.451795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.822230+0000","last_change":"2024-03-11T20:51:15.753932+0000","last_active":"2024-03-11T20:52:00.822230+0000","last_peered":"2024-03-11T20:52:00.822230+0000","last_clean":"2024-03-11T20:52:00.822230+0000","last_became_active":"2024-03-11T20:51:15.753549+0000","last_became_peered":"2024-03-11T20:51:15.753549+0000","last_unstale":"2024-03-11T20:52:00.822230+0000","last_undegraded":"2024-03-11T20:52:00.822230+0000","last_fullsized":"2024-03-11T20:52:00.822230+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:37:10.766409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.707404+0000","last_change":"2024-03-11T20:51:11.728497+0000","last_active":"2024-03-11T20:52:00.707404+0000","last_peered":"2024-03-11T20:52:00.707404+0000","last_clean":"2024-03-11T20:52:00.707404+0000","last_became_active":"2024-03-11T20:51:11.727489+0000","last_became_peered":"2024-03-11T20:51:11.727489+0000","last_unstale":"2024-03-11T20:52:00.707404+0000","last_undegraded":"2024-03-11T20:52:00.707404+0000","last_fullsized":"2024-03-11T20:52:00.707404+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:33:39.400474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'11","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709232+0000","last_change":"2024-03-11T20:51:13.735720+0000","last_active":"2024-03-11T20:52:00.709232+0000","last_peered":"2024-03-11T20:52:00.709232+0000","last_clean":"2024-03-11T20:52:00.709232+0000","last_became_active":"2024-03-11T20:51:13.734446+0000","last_became_peered":"2024-03-11T20:51:13.734446+0000","last_unstale":"2024-03-11T20:52:00.709232+0000","last_undegraded":"2024-03-11T20:52:00.709232+0000","last_fullsized":"2024-03-11T20:52:00.709232+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:28:44.865346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.708036+0000","last_change":"2024-03-11T20:51:17.755796+0000","last_active":"2024-03-11T20:52:00.708036+0000","last_peered":"2024-03-11T20:52:00.708036+0000","last_clean":"2024-03-11T20:52:00.708036+0000","last_became_active":"2024-03-11T20:51:17.755488+0000","last_became_peered":"2024-03-11T20:51:17.755488+0000","last_unstale":"2024-03-11T20:52:00.708036+0000","last_undegraded":"2024-03-11T20:52:00.708036+0000","last_fullsized":"2024-03-11T20:52:00.708036+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:58:24.604313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.345687+0000","last_change":"2024-03-11T20:51:15.743301+0000","last_active":"2024-03-11T20:52:01.345687+0000","last_peered":"2024-03-11T20:52:01.345687+0000","last_clean":"2024-03-11T20:52:01.345687+0000","last_became_active":"2024-03-11T20:51:15.742824+0000","last_became_peered":"2024-03-11T20:51:15.742824+0000","last_unstale":"2024-03-11T20:52:01.345687+0000","last_undegraded":"2024-03-11T20:52:01.345687+0000","last_fullsized":"2024-03-11T20:52:01.345687+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:05:00.074718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:01.343986+0000","last_change":"2024-03-11T20:51:11.731540+0000","last_active":"2024-03-11T20:52:01.343986+0000","last_peered":"2024-03-11T20:52:01.343986+0000","last_clean":"2024-03-11T20:52:01.343986+0000","last_became_active":"2024-03-11T20:51:11.731190+0000","last_became_peered":"2024-03-11T20:51:11.731190+0000","last_unstale":"2024-03-11T20:52:01.343986+0000","last_undegraded":"2024-03-11T20:52:01.343986+0000","last_fullsized":"2024-03-11T20:52:01.343986+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:32:05.461020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":57,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.708999+0000","last_change":"2024-03-11T20:51:13.736153+0000","last_active":"2024-03-11T20:52:00.708999+0000","last_peered":"2024-03-11T20:52:00.708999+0000","last_clean":"2024-03-11T20:52:00.708999+0000","last_became_active":"2024-03-11T20:51:13.734398+0000","last_became_peered":"2024-03-11T20:51:13.734398+0000","last_unstale":"2024-03-11T20:52:00.708999+0000","last_undegraded":"2024-03-11T20:52:00.708999+0000","last_fullsized":"2024-03-11T20:52:00.708999+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:30:13.152842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709585+0000","last_change":"2024-03-11T20:51:17.756970+0000","last_active":"2024-03-11T20:52:00.709585+0000","last_peered":"2024-03-11T20:52:00.709585+0000","last_clean":"2024-03-11T20:52:00.709585+0000","last_became_active":"2024-03-11T20:51:17.756656+0000","last_became_peered":"2024-03-11T20:51:17.756656+0000","last_unstale":"2024-03-11T20:52:00.709585+0000","last_undegraded":"2024-03-11T20:52:00.709585+0000","last_fullsized":"2024-03-11T20:52:00.709585+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:16.731560+0000","last_clean_scrub_stamp":"2024-03-11T20:51:16.731560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T22:30:16.480717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":48,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:16.788338+0000","last_change":"2024-03-11T20:51:15.752251+0000","last_active":"2024-03-11T20:52:16.788338+0000","last_peered":"2024-03-11T20:52:16.788338+0000","last_clean":"2024-03-11T20:52:16.788338+0000","last_became_active":"2024-03-11T20:51:15.751954+0000","last_became_peered":"2024-03-11T20:51:15.751954+0000","last_unstale":"2024-03-11T20:52:16.788338+0000","last_undegraded":"2024-03-11T20:52:16.788338+0000","last_fullsized":"2024-03-11T20:52:16.788338+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:14.719665+0000","last_clean_scrub_stamp":"2024-03-11T20:51:14.719665+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:53:52.008655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.710578+0000","last_change":"2024-03-11T20:51:11.731064+0000","last_active":"2024-03-11T20:52:00.710578+0000","last_peered":"2024-03-11T20:52:00.710578+0000","last_clean":"2024-03-11T20:52:00.710578+0000","last_became_active":"2024-03-11T20:51:11.729651+0000","last_became_peered":"2024-03-11T20:51:11.729651+0000","last_unstale":"2024-03-11T20:52:00.710578+0000","last_undegraded":"2024-03-11T20:52:00.710578+0000","last_fullsized":"2024-03-11T20:52:00.710578+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:10.695096+0000","last_clean_scrub_stamp":"2024-03-11T20:51:10.695096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:32:02.365119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":44,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-11T20:52:00.709276+0000","last_change":"2024-03-11T20:51:13.730282+0000","last_active":"2024-03-11T20:52:00.709276+0000","last_peered":"2024-03-11T20:52:00.709276+0000","last_clean":"2024-03-11T20:52:00.709276+0000","last_became_active":"2024-03-11T20:51:13.729916+0000","last_became_peered":"2024-03-11T20:51:13.729916+0000","last_unstale":"2024-03-11T20:52:00.709276+0000","last_undegraded":"2024-03-11T20:52:00.709276+0000","last_fullsized":"2024-03-11T20:52:00.709276+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T20:51:12.707892+0000","last_clean_scrub_stamp":"2024-03-11T20:51:12.707892+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T21:34:26.710640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":64,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28396,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716756,"statfs":{"total":95995035648,"available":95965958144,"internally_reserved":0,"allocated":1409024,"data_stored":929902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495642,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28368,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716784,"statfs":{"total":95995035648,"available":95965986816,"internally_reserved":0,"allocated":1384448,"data_stored":928362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32004,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713148,"statfs":{"total":95995035648,"available":95962263552,"internally_reserved":0,"allocated":843776,"data_stored":339026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28396,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716756,"statfs":{"total":95995035648,"available":95965958144,"internally_reserved":0,"allocated":1413120,"data_stored":929644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":794624,"data_stored":338003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182439,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27828,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717324,"statfs":{"total":95995035648,"available":95966539776,"internally_reserved":0,"allocated":827392,"data_stored":338885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018915,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":819200,"data_stored":339074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":794624,"data_stored":338320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T20:52:32.756 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T20:52:32.756 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T20:52:32.756 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T20:52:32.756 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T20:52:32.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:32 smithi129 ceph-mon[30184]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:52:32.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:32 smithi129 ceph-mon[42066]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:52:33.003 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T20:52:33.202 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:32 smithi157 ceph-mon[33359]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T20:52:33.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:52:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:52:34.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:33 smithi129 ceph-mon[30184]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:34.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:33 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:34.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:33 smithi129 ceph-mon[42066]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:34.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:33 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:33 smithi157 ceph-mon[33359]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:33 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:36.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:35 smithi129 ceph-mon[30184]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:36.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:35 smithi129 ceph-mon[42066]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:36.229 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:52:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:52:36.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:35 smithi157 ceph-mon[33359]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:37.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.744+0000 7f95893f5700 1 -- 172.21.15.129:0/1459594140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9584101e90 msgr2=0x7f9584102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:37.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.744+0000 7f95893f5700 1 --2- 172.21.15.129:0/1459594140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9584101e90 0x7f9584102310 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f956c009990 tx=0x7f956c009ca0 comp rx=0 tx=0).stop 2024-03-11T20:52:37.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.744+0000 7f95893f5700 1 -- 172.21.15.129:0/1459594140 shutdown_connections 2024-03-11T20:52:37.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.744+0000 7f95893f5700 1 --2- 172.21.15.129:0/1459594140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9584102850 0x7f958410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:37.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.744+0000 7f95893f5700 1 --2- 172.21.15.129:0/1459594140 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9584107f00 0x7f95841082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:37.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.744+0000 7f95893f5700 1 --2- 172.21.15.129:0/1459594140 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9584101e90 0x7f9584102310 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:37.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.744+0000 7f95893f5700 1 -- 172.21.15.129:0/1459594140 >> 172.21.15.129:0/1459594140 conn(0x7f9584078eb0 msgr2=0x7f95840792c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:37.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.744+0000 7f95893f5700 1 -- 172.21.15.129:0/1459594140 shutdown_connections 2024-03-11T20:52:37.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.744+0000 7f95893f5700 1 -- 172.21.15.129:0/1459594140 wait complete. 2024-03-11T20:52:37.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.745+0000 7f95893f5700 1 Processor -- start 2024-03-11T20:52:37.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.745+0000 7f95893f5700 1 -- start start 2024-03-11T20:52:37.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.745+0000 7f95893f5700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9584101e90 0x7f958419f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:37.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.745+0000 7f95893f5700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9584102850 0x7f958419fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:37.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.745+0000 7f95893f5700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9584107f00 0x7f9584199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:37.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.745+0000 7f95893f5700 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9584111610 con 0x7f9584101e90 2024-03-11T20:52:37.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.745+0000 7f95893f5700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9584111490 con 0x7f9584102850 2024-03-11T20:52:37.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.745+0000 7f95893f5700 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9584111790 con 0x7f9584107f00 2024-03-11T20:52:37.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.745+0000 7f95837fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9584107f00 0x7f9584199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:37.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.745+0000 7f9582ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9584101e90 0x7f958419f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:37.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.745+0000 7f95837fe700 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9584107f00 0x7f9584199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:35396/0 (socket says 172.21.15.129:35396) 2024-03-11T20:52:37.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.745+0000 7f9582ffd700 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9584101e90 0x7f958419f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45110/0 (socket says 172.21.15.129:45110) 2024-03-11T20:52:37.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.745+0000 7f95837fe700 1 -- 172.21.15.129:0/1385677754 learned_addr learned my addr 172.21.15.129:0/1385677754 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T20:52:37.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.746+0000 7f95827fc700 1 --2- 172.21.15.129:0/1385677754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9584102850 0x7f958419fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:37.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.746+0000 7f95837fe700 1 -- 172.21.15.129:0/1385677754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9584102850 msgr2=0x7f958419fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:37.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.746+0000 7f95837fe700 1 --2- 172.21.15.129:0/1385677754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9584102850 0x7f958419fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:37.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.746+0000 7f95837fe700 1 -- 172.21.15.129:0/1385677754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9584101e90 msgr2=0x7f958419f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:37.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.746+0000 7f95837fe700 1 --2- 172.21.15.129:0/1385677754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9584101e90 0x7f958419f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:37.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.746+0000 7f95837fe700 1 -- 172.21.15.129:0/1385677754 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958419a260 con 0x7f9584107f00 2024-03-11T20:52:37.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.746+0000 7f9582ffd700 1 --2- 172.21.15.129:0/1385677754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9584101e90 0x7f958419f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T20:52:37.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.746+0000 7f95827fc700 1 --2- 172.21.15.129:0/1385677754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9584102850 0x7f958419fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T20:52:37.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.746+0000 7f95837fe700 1 --2- 172.21.15.129:0/1385677754 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9584107f00 0x7f9584199b00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f957c00d980 tx=0x7f957c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:37.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.746+0000 7f957bfff700 1 -- 172.21.15.129:0/1385677754 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c017070 con 0x7f9584107f00 2024-03-11T20:52:37.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.746+0000 7f957bfff700 1 -- 172.21.15.129:0/1385677754 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f957c005020 con 0x7f9584107f00 2024-03-11T20:52:37.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.746+0000 7f95893f5700 1 -- 172.21.15.129:0/1385677754 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958419a550 con 0x7f9584107f00 2024-03-11T20:52:37.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.746+0000 7f957bfff700 1 -- 172.21.15.129:0/1385677754 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c009990 con 0x7f9584107f00 2024-03-11T20:52:37.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.746+0000 7f95893f5700 1 -- 172.21.15.129:0/1385677754 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95841a67c0 con 0x7f9584107f00 2024-03-11T20:52:37.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.747+0000 7f95893f5700 1 -- 172.21.15.129:0/1385677754 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9584068aa0 con 0x7f9584107f00 2024-03-11T20:52:37.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.748+0000 7f957bfff700 1 -- 172.21.15.129:0/1385677754 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f957c009b30 con 0x7f9584107f00 2024-03-11T20:52:37.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.748+0000 7f957bfff700 1 --2- 172.21.15.129:0/1385677754 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f957006de80 0x7f9570070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T20:52:37.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.748+0000 7f957bfff700 1 -- 172.21.15.129:0/1385677754 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7508+0+0 (secure 0 0 0) 0x7f957c08e030 con 0x7f9584107f00 2024-03-11T20:52:37.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.748+0000 7f9582ffd700 1 --2- 172.21.15.129:0/1385677754 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f957006de80 0x7f9570070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T20:52:37.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.749+0000 7f9582ffd700 1 --2- 172.21.15.129:0/1385677754 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f957006de80 0x7f9570070340 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9584079520 tx=0x7f957400b040 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T20:52:37.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:37.753+0000 7f957bfff700 1 -- 172.21.15.129:0/1385677754 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f957c020020 con 0x7f9584107f00 2024-03-11T20:52:38.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:38.005+0000 7f95893f5700 1 -- 172.21.15.129:0/1385677754 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f95841a69f0 con 0x7f9584107f00 2024-03-11T20:52:38.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:38.005+0000 7f957bfff700 1 -- 172.21.15.129:0/1385677754 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f957c009e20 con 0x7f9584107f00 2024-03-11T20:52:38.002 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-03-11T20:52:38.002 INFO:teuthology.orchestra.run.smithi129.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T20:52:38.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:38.007+0000 7f95893f5700 1 -- 172.21.15.129:0/1385677754 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f957006de80 msgr2=0x7f9570070340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:38.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:38.007+0000 7f95893f5700 1 --2- 172.21.15.129:0/1385677754 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f957006de80 0x7f9570070340 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9584079520 tx=0x7f957400b040 comp rx=0 tx=0).stop 2024-03-11T20:52:38.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:38.007+0000 7f95893f5700 1 -- 172.21.15.129:0/1385677754 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9584107f00 msgr2=0x7f9584199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T20:52:38.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:38.007+0000 7f95893f5700 1 --2- 172.21.15.129:0/1385677754 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9584107f00 0x7f9584199b00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f957c00d980 tx=0x7f957c00dc90 comp rx=0 tx=0).stop 2024-03-11T20:52:38.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:38.007+0000 7f95893f5700 1 -- 172.21.15.129:0/1385677754 shutdown_connections 2024-03-11T20:52:38.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:38.007+0000 7f95893f5700 1 --2- 172.21.15.129:0/1385677754 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f957006de80 0x7f9570070340 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:38.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:38.007+0000 7f95893f5700 1 --2- 172.21.15.129:0/1385677754 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9584107f00 0x7f9584199b00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:38.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:38.007+0000 7f95893f5700 1 --2- 172.21.15.129:0/1385677754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9584101e90 0x7f958419f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:38.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:38.007+0000 7f95893f5700 1 --2- 172.21.15.129:0/1385677754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9584102850 0x7f958419fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T20:52:38.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:38.007+0000 7f95893f5700 1 -- 172.21.15.129:0/1385677754 >> 172.21.15.129:0/1385677754 conn(0x7f9584078eb0 msgr2=0x7f958410cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T20:52:38.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:38.007+0000 7f95893f5700 1 -- 172.21.15.129:0/1385677754 shutdown_connections 2024-03-11T20:52:38.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-03-11T20:52:38.008+0000 7f95893f5700 1 -- 172.21.15.129:0/1385677754 wait complete. 2024-03-11T20:52:38.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:37 smithi129 ceph-mon[30184]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:38.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:37 smithi129 ceph-mon[42066]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:38.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:37 smithi157 ceph-mon[33359]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:39.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1385677754' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T20:52:39.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1385677754' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T20:52:39.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1385677754' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T20:52:39.351 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T20:52:39.351 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T20:52:39.352 INFO:teuthology.run_tasks:Running task workunit... 2024-03-11T20:52:39.366 INFO:tasks.workunit:Pulling workunits from ref 1c349451176cc5b4ebfb24b22eaaa754e05cff6c 2024-03-11T20:52:39.366 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-11T20:52:39.367 DEBUG:teuthology.orchestra.run.smithi129:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-11T20:52:39.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T20:52:39.383 INFO:teuthology.orchestra.run.smithi129.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-11T20:52:39.383 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-11T20:52:39.440 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-11T20:52:39.440 DEBUG:teuthology.orchestra.run.smithi129:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-11T20:52:39.494 INFO:tasks.workunit:timeout=3h 2024-03-11T20:52:39.494 INFO:tasks.workunit:cleanup=True 2024-03-11T20:52:39.495 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1c349451176cc5b4ebfb24b22eaaa754e05cff6c 2024-03-11T20:52:39.549 INFO:tasks.workunit.client.0.smithi129.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-11T20:52:39.601 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:39 smithi129 podman[79851]: 2024-03-11 20:52:39.452713332 +0000 UTC m=+0.018609744 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-11T20:52:39.601 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:39 smithi129 podman[79851]: 2024-03-11 20:52:39.6059097 +0000 UTC m=+0.171806105 volume create ed93668f4cd9ef0bff0559f85bb97d96ecbe7d689269db5f2eed4f6e27914425 2024-03-11T20:52:39.881 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:39 smithi129 podman[79851]: 2024-03-11 20:52:39.698339841 +0000 UTC m=+0.264236243 container create 6ec5f38dae02cd027a42c34df1ca29799a9b332a755d8e9d9bec9a87ba6191ee (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T20:52:40.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:39 smithi129 ceph-mon[30184]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:40.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:39 smithi129 ceph-mon[42066]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:40.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:39 smithi129 podman[79851]: 2024-03-11 20:52:39.993053579 +0000 UTC m=+0.558949990 container init 6ec5f38dae02cd027a42c34df1ca29799a9b332a755d8e9d9bec9a87ba6191ee (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T20:52:40.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:39 smithi129 podman[79851]: 2024-03-11 20:52:39.998570363 +0000 UTC m=+0.564466768 container start 6ec5f38dae02cd027a42c34df1ca29799a9b332a755d8e9d9bec9a87ba6191ee (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T20:52:40.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:40 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[79919]: ts=2024-03-11T20:52:40.010Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-11T20:52:40.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:40 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[79919]: ts=2024-03-11T20:52:40.011Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-11T20:52:40.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:40 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[79919]: ts=2024-03-11T20:52:40.011Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.129 port=9094 2024-03-11T20:52:40.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:40 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[79919]: ts=2024-03-11T20:52:40.012Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T20:52:40.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:40 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[79919]: ts=2024-03-11T20:52:40.036Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T20:52:40.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:40 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[79919]: ts=2024-03-11T20:52:40.036Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T20:52:40.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:40 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[79919]: ts=2024-03-11T20:52:40.039Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-11T20:52:40.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:40 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[79919]: ts=2024-03-11T20:52:40.039Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-11T20:52:40.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:40 smithi129 bash[79851]: 6ec5f38dae02cd027a42c34df1ca29799a9b332a755d8e9d9bec9a87ba6191ee 2024-03-11T20:52:40.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:40 smithi129 systemd[1]: Started Ceph alertmanager.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T20:52:40.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:39 smithi157 ceph-mon[33359]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:40.728 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:40 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:52:40] ENGINE Bus STOPPING 2024-03-11T20:52:40.728 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:40 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:52:40] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T20:52:40.728 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:40 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:52:40] ENGINE Bus STOPPED 2024-03-11T20:52:40.729 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:40 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:52:40] ENGINE Bus STARTING 2024-03-11T20:52:40.729 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:40 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:52:40] ENGINE Serving on http://:::9283 2024-03-11T20:52:40.729 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:40 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:52:40] ENGINE Bus STARTED 2024-03-11T20:52:41.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[42066]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T20:52:41.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T20:52:41.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[42066]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T20:52:41.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[42066]: Deploying daemon grafana.a on smithi157 2024-03-11T20:52:41.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[30184]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T20:52:41.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T20:52:41.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[30184]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T20:52:41.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[30184]: Deploying daemon grafana.a on smithi157 2024-03-11T20:52:41.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:41 smithi157 ceph-mon[33359]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T20:52:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T20:52:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:41 smithi157 ceph-mon[33359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T20:52:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:41 smithi157 ceph-mon[33359]: Deploying daemon grafana.a on smithi157 2024-03-11T20:52:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:42.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:42 smithi129 ceph-mon[30184]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:42.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:42 smithi129 ceph-mon[42066]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:42.479 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:42 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[79919]: ts=2024-03-11T20:52:42.013Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000341295s 2024-03-11T20:52:42.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:42 smithi157 ceph-mon[33359]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:43.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:52:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:52:44.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:44 smithi157 ceph-mon[33359]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:44.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:44 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:44.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:44 smithi129 ceph-mon[30184]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:44.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:44 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:44.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:44 smithi129 ceph-mon[42066]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:44.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:44 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:46.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:46 smithi129 ceph-mon[30184]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:46.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:46 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:46.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:52:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:52:46.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:46 smithi129 ceph-mon[42066]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:46.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:46 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:46.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:46 smithi157 ceph-mon[33359]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:46.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:46 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:52:48.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:48 smithi129 ceph-mon[30184]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:48.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:48 smithi129 ceph-mon[42066]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:48.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:48 smithi157 ceph-mon[33359]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:50.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:50 smithi129 ceph-mon[30184]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:50.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:50 smithi129 ceph-mon[42066]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:50.479 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:52:50 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[79919]: ts=2024-03-11T20:52:50.016Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004174688s 2024-03-11T20:52:50.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:50 smithi157 ceph-mon[33359]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:52.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:52 smithi157 ceph-mon[33359]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:52.558 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:52 smithi129 ceph-mon[30184]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:52.558 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:52 smithi129 ceph-mon[42066]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:53.447 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:52:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:52:54.588 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:54 smithi157 ceph-mon[33359]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:54.588 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:54 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:54.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:54 smithi129 ceph-mon[30184]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:54.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:54 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:54.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:54 smithi129 ceph-mon[42066]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:54.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:54 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:52:55.236 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 systemd[1]: Starting Ceph grafana.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T20:52:55.678 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 podman[66274]: 2024-03-11 20:52:55.205301294 +0000 UTC m=+0.016950405 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-11T20:52:55.678 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 podman[66274]: 2024-03-11 20:52:55.311724576 +0000 UTC m=+0.123373681 container create e2ff54303d9a8f0beafedddc055d08c8be8cb37836b9b98e4c759b984ca07733 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, description=Ceph Grafana Container, vendor=Red Hat, Inc., architecture=x86_64, distribution-scope=public, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-11T20:52:56.052 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 podman[66274]: 2024-03-11 20:52:55.806297747 +0000 UTC m=+0.617946845 container init e2ff54303d9a8f0beafedddc055d08c8be8cb37836b9b98e4c759b984ca07733 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a, version=8.5, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, release=239.1651231664, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, architecture=x86_64, distribution-scope=public, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, com.redhat.component=ubi8-container, io.buildah.version=1.32.0) 2024-03-11T20:52:56.052 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 podman[66274]: 2024-03-11 20:52:55.812240451 +0000 UTC m=+0.623889550 container start e2ff54303d9a8f0beafedddc055d08c8be8cb37836b9b98e4c759b984ca07733 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, description=Ceph Grafana Container, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, release=239.1651231664, maintainer=Paul Cuzner , distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897) 2024-03-11T20:52:56.052 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 bash[66274]: e2ff54303d9a8f0beafedddc055d08c8be8cb37836b9b98e4c759b984ca07733 2024-03-11T20:52:56.052 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 systemd[1]: Started Ceph grafana.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T20:52:56.052 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-11T20:52:55.966350215Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-11T20:52:56.052 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-11T20:52:55.966542299Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-11T20:52:56.052 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-11T20:52:55.966561778Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-11T20:52:56.052 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-11T20:52:55.966568932Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-11T20:52:56.053 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-11T20:52:55.966575105Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-11T20:52:56.053 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-11T20:52:55.966581135Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-11T20:52:56.053 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-11T20:52:55.966586826Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-11T20:52:56.053 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-11T20:52:55.96659257Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-11T20:52:56.053 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-11T20:52:55.966598447Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-11T20:52:56.053 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-11T20:52:55.96660414Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-11T20:52:56.053 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-11T20:52:55.966610649Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-11T20:52:56.053 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-11T20:52:55.96661628Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-11T20:52:56.053 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-11T20:52:55.966621898Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-11T20:52:56.053 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-11T20:52:55.966627769Z level=info msg="App mode production" 2024-03-11T20:52:56.054 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=sqlstore t=2024-03-11T20:52:55.966672557Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-11T20:52:56.054 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=sqlstore t=2024-03-11T20:52:55.966689019Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-11T20:52:56.054 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.967546999Z level=info msg="Starting DB migrations" 2024-03-11T20:52:56.054 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.967636059Z level=info msg="Executing migration" id="create migration_log table" 2024-03-11T20:52:56.054 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.968149989Z level=info msg="Executing migration" id="create user table" 2024-03-11T20:52:56.054 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.968546793Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-11T20:52:56.054 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.96898473Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-11T20:52:56.054 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.969368532Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-11T20:52:56.054 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.96987886Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-11T20:52:56.055 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.970244883Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-11T20:52:56.055 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.970917525Z level=info msg="Executing migration" id="create user table v2" 2024-03-11T20:52:56.055 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.971307148Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-11T20:52:56.055 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.971652483Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-11T20:52:56.055 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.972064286Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-11T20:52:56.055 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.972383412Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-11T20:52:56.055 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.972669488Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-11T20:52:56.055 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.973015627Z level=info msg="Executing migration" id="Update user table charset" 2024-03-11T20:52:56.056 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.973276458Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-11T20:52:56.056 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.973618933Z level=info msg="Executing migration" id="Add missing user data" 2024-03-11T20:52:56.056 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.973897972Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-11T20:52:56.056 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.974287958Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-11T20:52:56.056 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.974617863Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-11T20:52:56.056 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.974970609Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-11T20:52:56.056 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.976051861Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-11T20:52:56.056 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.976446095Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-11T20:52:56.056 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.976844869Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-11T20:52:56.056 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.977236981Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-11T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.977627662Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-11T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.978013064Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-11T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.97827494Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-11T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.978601512Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-11T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.978928598Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-11T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.97929957Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-11T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.979628098Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-11T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.980263898Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-11T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.980654427Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-11T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.981029065Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-11T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.98141563Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-11T20:52:56.057 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.981784373Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-11T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.982206563Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-11T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.982506896Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-11T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.982787305Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-11T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.983101122Z level=info msg="Executing migration" id="create star table" 2024-03-11T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.983435141Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-11T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.983823451Z level=info msg="Executing migration" id="create org table v1" 2024-03-11T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.984238661Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-11T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.984622666Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-11T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.984981507Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-11T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.985391162Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-11T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.985796703Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-11T20:52:56.058 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.986171777Z level=info msg="Executing migration" id="Update org table charset" 2024-03-11T20:52:56.059 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.986426054Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-11T20:52:56.059 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.986653609Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-11T20:52:56.059 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.986907387Z level=info msg="Executing migration" id="create dashboard table" 2024-03-11T20:52:56.059 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.987256911Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-11T20:52:56.059 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.987635295Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-11T20:52:56.059 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.98801351Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-11T20:52:56.059 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.98839501Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-11T20:52:56.059 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.988806176Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-11T20:52:56.059 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.989189976Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-11T20:52:56.059 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.990019736Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-11T20:52:56.059 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.990384364Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-11T20:52:56.060 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.990767342Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-11T20:52:56.060 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.991180672Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-11T20:52:56.060 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.991495399Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-11T20:52:56.060 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.991821589Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-11T20:52:56.060 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.992097799Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-11T20:52:56.060 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.992539039Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-11T20:52:56.062 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.992949963Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-11T20:52:56.062 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.993423946Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-11T20:52:56.062 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.99378316Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-11T20:52:56.062 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.994288401Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-11T20:52:56.062 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.994682745Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-11T20:52:56.062 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.995076755Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-11T20:52:56.062 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.995344406Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-11T20:52:56.062 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.99558303Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-11T20:52:56.063 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.9960637Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-11T20:52:56.063 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.99651753Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-11T20:52:56.063 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.996992725Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-11T20:52:56.063 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.997451451Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-11T20:52:56.063 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.997714903Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-11T20:52:56.063 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.998139206Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-11T20:52:56.063 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.99850865Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-11T20:52:56.063 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.998744606Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-11T20:52:56.063 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.99915152Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-11T20:52:56.063 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:55.999501714Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-11T20:52:56.063 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.000420588Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-11T20:52:56.063 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.000794705Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-11T20:52:56.064 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.001221647Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-11T20:52:56.064 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.001616639Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-11T20:52:56.064 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.001886614Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-11T20:52:56.064 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.00221582Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-11T20:52:56.064 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.002715686Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-11T20:52:56.064 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.003175362Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-11T20:52:56.064 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.003460838Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-11T20:52:56.064 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.003711702Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-11T20:52:56.064 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.004099779Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-11T20:52:56.064 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.00459506Z level=info msg="Executing migration" id="create data_source table" 2024-03-11T20:52:56.065 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.00498743Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-11T20:52:56.065 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.005397337Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-11T20:52:56.065 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.005791515Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-11T20:52:56.065 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.006169992Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-11T20:52:56.065 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.006549236Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-11T20:52:56.065 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.007557015Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-11T20:52:56.065 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.00796578Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-11T20:52:56.065 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.008373785Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-11T20:52:56.065 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.008773391Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-11T20:52:56.065 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.009113635Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-11T20:52:56.065 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.009605473Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-11T20:52:56.066 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.010106714Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-11T20:52:56.066 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.010369879Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-11T20:52:56.066 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.010631833Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-11T20:52:56.066 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.01115969Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-11T20:52:56.066 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.011447156Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-11T20:52:56.066 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.011702798Z level=info msg="Executing migration" id="Add uid column" 2024-03-11T20:52:56.066 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.012214398Z level=info msg="Executing migration" id="Update uid value" 2024-03-11T20:52:56.066 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.012503651Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-11T20:52:56.066 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.012870882Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-11T20:52:56.066 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.013284139Z level=info msg="Executing migration" id="create api_key table" 2024-03-11T20:52:56.066 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.013683536Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-11T20:52:56.067 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.014083467Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-11T20:52:56.067 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.01446822Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-11T20:52:56.067 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.014888464Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-11T20:52:56.067 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.015314158Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-11T20:52:56.067 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.01569723Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-11T20:52:56.067 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.01612292Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-11T20:52:56.067 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.017122108Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-11T20:52:56.067 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.017508193Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-11T20:52:56.067 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.01787136Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-11T20:52:56.067 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.018285187Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-11T20:52:56.068 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.018692597Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-11T20:52:56.068 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.018996774Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-11T20:52:56.068 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.019324521Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-11T20:52:56.068 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.019559914Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-11T20:52:56.068 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.02007713Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-11T20:52:56.068 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.020578445Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-11T20:52:56.068 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.020837812Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-11T20:52:56.068 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.021373503Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-11T20:52:56.068 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.021886664Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-11T20:52:56.068 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.022255643Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-11T20:52:56.068 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.022568432Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-11T20:52:56.069 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.022898467Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-11T20:52:56.069 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.023298303Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-11T20:52:56.069 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.023681516Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-11T20:52:56.069 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.024077649Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-11T20:52:56.069 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.024347746Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-11T20:52:56.069 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.02458211Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-11T20:52:56.069 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.025123295Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-11T20:52:56.069 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.025645249Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-11T20:52:56.069 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.025915482Z level=info msg="Executing migration" id="create quota table v1" 2024-03-11T20:52:56.069 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.026302197Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-11T20:52:56.069 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.026714873Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-11T20:52:56.070 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.026998689Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-11T20:52:56.070 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.027386421Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-11T20:52:56.070 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.027807197Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-11T20:52:56.070 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.02835225Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-11T20:52:56.070 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.028626173Z level=info msg="Executing migration" id="create session table" 2024-03-11T20:52:56.070 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.029028329Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-11T20:52:56.070 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.02931349Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-11T20:52:56.070 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.029559977Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-11T20:52:56.070 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.02993907Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-11T20:52:56.070 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.030321875Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-11T20:52:56.071 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.030551977Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-11T20:52:56.071 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.030791667Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-11T20:52:56.071 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.031065544Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-11T20:52:56.071 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.03135403Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-11T20:52:56.071 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.031740082Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-11T20:52:56.071 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.031991218Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-11T20:52:56.071 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.032612163Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-11T20:52:56.071 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.032873724Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-11T20:52:56.071 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.033469781Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-11T20:52:56.071 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.03406776Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-11T20:52:56.072 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.034372353Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-11T20:52:56.072 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.034773518Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-11T20:52:56.072 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.035241601Z level=info msg="Executing migration" id="create alert table v1" 2024-03-11T20:52:56.072 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.035684127Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-11T20:52:56.072 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.036171666Z level=info msg="Executing migration" id="add index alert state" 2024-03-11T20:52:56.072 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.036591192Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-11T20:52:56.072 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.036978714Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-11T20:52:56.072 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.037357948Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-11T20:52:56.072 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.03777888Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-11T20:52:56.072 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.038244071Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-11T20:52:56.072 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.039638933Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-11T20:52:56.073 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.039998374Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-11T20:52:56.073 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.040471346Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-11T20:52:56.073 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.040737972Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-11T20:52:56.073 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.041087599Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-11T20:52:56.073 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.041452175Z level=info msg="Executing migration" id="Add column is_default" 2024-03-11T20:52:56.073 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.042094302Z level=info msg="Executing migration" id="Add column frequency" 2024-03-11T20:52:56.073 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.042709535Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-11T20:52:56.073 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.043382745Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-11T20:52:56.073 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.044037504Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-11T20:52:56.073 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.044445875Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-11T20:52:56.073 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.044687105Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-11T20:52:56.073 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.044925706Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-11T20:52:56.073 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.045335814Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-11T20:52:56.074 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.045809074Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-11T20:52:56.074 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.046194328Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-11T20:52:56.074 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.046552551Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-11T20:52:56.074 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.046913124Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-11T20:52:56.074 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.047529511Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-11T20:52:56.074 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.048128317Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-11T20:52:56.074 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.048415771Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-11T20:52:56.074 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.048814184Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-11T20:52:56.074 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.049195879Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-11T20:52:56.074 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.049776573Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-11T20:52:56.074 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.050022658Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-11T20:52:56.074 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.050490397Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-11T20:52:56.074 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.051042823Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-11T20:52:56.074 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.051368979Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-11T20:52:56.075 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.051804134Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-11T20:52:56.075 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.052236902Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-11T20:52:56.075 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.052663994Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-11T20:52:56.075 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.053091871Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-11T20:52:56.075 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.053544651Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-11T20:52:56.075 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.054030825Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-11T20:52:56.075 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.054321259Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-11T20:52:56.075 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.054946305Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-11T20:52:56.075 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.055358329Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-11T20:52:56.075 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.055977735Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-11T20:52:56.075 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.056377209Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-11T20:52:56.075 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.056941817Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-11T20:52:56.075 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.057511509Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-11T20:52:56.075 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.060085635Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-11T20:52:56.076 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.060588115Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-11T20:52:56.076 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.061157136Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-11T20:52:56.076 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.061559127Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-11T20:52:56.076 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.061985189Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-11T20:52:56.076 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.062366175Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-11T20:52:56.076 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.063357546Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-11T20:52:56.076 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.064337341Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-11T20:52:56.076 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.064876144Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-11T20:52:56.076 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.0654341Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-11T20:52:56.076 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.065821976Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-11T20:52:56.076 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.067021744Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-11T20:52:56.478 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:52:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:52:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:52:56.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:56 smithi129 ceph-mon[30184]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:56.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:56 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:56.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:56 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:56.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:56 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:56.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:56 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:56.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:56 smithi129 ceph-mon[42066]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:56.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:56 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:56.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:56 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:56.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:56 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:56.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:56 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:56.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-mon[33359]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:52:56.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:56.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:56.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:56.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:52:56.486 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.067490138Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-11T20:52:56.486 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.067733424Z level=info msg="Executing migration" id="Move region to single row" 2024-03-11T20:52:56.486 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.068067345Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-11T20:52:56.486 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.068452795Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-11T20:52:56.486 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.068792155Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-11T20:52:56.486 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.069290879Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-11T20:52:56.487 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.069697841Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-11T20:52:56.487 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.070067917Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-11T20:52:56.487 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.070437119Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-11T20:52:56.487 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.070680336Z level=info msg="Executing migration" id="create test_data table" 2024-03-11T20:52:56.487 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.071087346Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-11T20:52:56.487 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.071442913Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-11T20:52:56.487 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.071822431Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-11T20:52:56.487 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.072217324Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-11T20:52:56.487 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.072494476Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-11T20:52:56.487 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.072756295Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-11T20:52:56.487 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.072988002Z level=info msg="Executing migration" id="create team table" 2024-03-11T20:52:56.487 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.07334153Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-11T20:52:56.488 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.073763592Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-11T20:52:56.488 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.074156063Z level=info msg="Executing migration" id="create team member table" 2024-03-11T20:52:56.488 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.074511872Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-11T20:52:56.488 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.074889344Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-11T20:52:56.488 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.075281699Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-11T20:52:56.488 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.075659801Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-11T20:52:56.488 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.07636Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-11T20:52:56.488 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.077039526Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-11T20:52:56.488 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.077753155Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-11T20:52:56.488 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.078129137Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-11T20:52:56.489 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.078546623Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-11T20:52:56.489 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.078945278Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-11T20:52:56.489 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.079338598Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-11T20:52:56.489 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.079796158Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-11T20:52:56.489 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.080184532Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-11T20:52:56.489 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.080573558Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-11T20:52:56.489 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.080942078Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-11T20:52:56.489 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.081327764Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-11T20:52:56.489 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.081577982Z level=info msg="Executing migration" id="create tag table" 2024-03-11T20:52:56.489 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.081912482Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-11T20:52:56.489 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.082326751Z level=info msg="Executing migration" id="create login attempt table" 2024-03-11T20:52:56.489 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.082688709Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-11T20:52:56.489 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.083075278Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-11T20:52:56.489 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.083472876Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-11T20:52:56.490 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.085319005Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-11T20:52:56.490 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.085691757Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-11T20:52:56.490 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.086084141Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-11T20:52:56.490 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.086393911Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-11T20:52:56.490 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.086684451Z level=info msg="Executing migration" id="create user auth table" 2024-03-11T20:52:56.490 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.08700331Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-11T20:52:56.490 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.08740898Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-11T20:52:56.490 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.087634081Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-11T20:52:56.490 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.088346207Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-11T20:52:56.490 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.089078673Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-11T20:52:56.490 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.089798541Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-11T20:52:56.490 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.090596Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-11T20:52:56.490 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.091058533Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-11T20:52:56.490 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.091819349Z level=info msg="Executing migration" id="create server_lock table" 2024-03-11T20:52:56.491 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.092187631Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-11T20:52:56.492 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.092618637Z level=info msg="Executing migration" id="create user auth token table" 2024-03-11T20:52:56.492 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.093016609Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-11T20:52:56.492 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.093435633Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-11T20:52:56.492 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.093845522Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-11T20:52:56.492 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.094241315Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-11T20:52:56.492 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.094952151Z level=info msg="Executing migration" id="create cache_data table" 2024-03-11T20:52:56.492 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.095339631Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-11T20:52:56.492 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.095728234Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-11T20:52:56.492 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.096122344Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-11T20:52:56.492 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.096528591Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-11T20:52:56.493 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.096757018Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-11T20:52:56.493 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.097030891Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-11T20:52:56.493 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.097460245Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-11T20:52:56.493 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.097871653Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-11T20:52:56.493 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.098261675Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-11T20:52:56.493 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.098547011Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-11T20:52:56.493 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.098885456Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-11T20:52:56.493 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.099269635Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-11T20:52:56.493 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.099655673Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-11T20:52:56.493 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.100048441Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-11T20:52:56.493 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.100808784Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-11T20:52:56.493 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.101207537Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-11T20:52:56.493 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.101497492Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-11T20:52:56.493 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.101830321Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-11T20:52:56.494 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.102259645Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-11T20:52:56.494 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.102663565Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-11T20:52:56.494 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.102893731Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-11T20:52:56.494 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.103289126Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-11T20:52:56.494 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.103687257Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-11T20:52:56.494 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.104101243Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-11T20:52:56.494 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.104516273Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-11T20:52:56.494 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.105263908Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-11T20:52:56.494 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.105659122Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-11T20:52:56.494 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.1060274Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-11T20:52:56.494 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.109310872Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-11T20:52:56.494 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.112644409Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-11T20:52:56.494 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.113057989Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-11T20:52:56.494 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.113464814Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-11T20:52:56.495 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.114245878Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-11T20:52:56.495 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.114648685Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-11T20:52:56.495 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.115061997Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-11T20:52:56.495 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.115476902Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-11T20:52:56.495 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.115878899Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-11T20:52:56.495 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.11614193Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-11T20:52:56.495 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.116945116Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-11T20:52:56.495 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.117747689Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-11T20:52:56.495 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.11854318Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-11T20:52:56.495 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.11892085Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-11T20:52:56.495 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.119325321Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-11T20:52:56.495 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.120140053Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-11T20:52:56.495 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.12091336Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-11T20:52:56.496 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.121320188Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-11T20:52:56.496 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.122092604Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-11T20:52:56.496 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.122904325Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-11T20:52:56.496 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.12321368Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-11T20:52:56.496 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.123642169Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-11T20:52:56.496 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.124090691Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-11T20:52:56.496 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.124564095Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-11T20:52:56.496 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.124806751Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-11T20:52:56.496 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.125699065Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-11T20:52:56.496 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.126530892Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-11T20:52:56.496 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.127361441Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-11T20:52:56.496 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.128171608Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-11T20:52:56.496 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.129002932Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-11T20:52:56.496 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.129306276Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-11T20:52:56.496 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.12966903Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-11T20:52:56.497 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.130525806Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-11T20:52:56.497 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.130798364Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-11T20:52:56.497 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.131668823Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-11T20:52:56.497 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.132097367Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-11T20:52:56.497 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.132896605Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-11T20:52:56.497 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.133281157Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-11T20:52:56.497 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.133728331Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-11T20:52:56.497 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.134589476Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-11T20:52:56.497 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.134980023Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-11T20:52:56.497 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.135423006Z level=info msg="Executing migration" id="create alert_image table" 2024-03-11T20:52:56.497 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.135804137Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-11T20:52:56.497 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.136226577Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-11T20:52:56.497 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.136541164Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-11T20:52:56.497 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.136924011Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-11T20:52:56.498 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.137336391Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T20:52:56.498 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.137420586Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T20:52:56.498 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.137638763Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-11T20:52:56.498 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.13794831Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-11T20:52:56.498 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.138377244Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-11T20:52:56.498 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.138506798Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-11T20:52:56.498 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.138615367Z level=info msg="alerts found to migrate" alerts=0 2024-03-11T20:52:56.498 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.138978913Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-11T20:52:56.498 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.139401245Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-11T20:52:56.498 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.139825947Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-11T20:52:56.498 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.140232542Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-11T20:52:56.498 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.140693662Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-11T20:52:56.498 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.141141564Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-11T20:52:56.498 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.141407353Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-11T20:52:56.499 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.141701867Z level=info msg="Executing migration" id="create data_keys table" 2024-03-11T20:52:56.499 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.142086701Z level=info msg="Executing migration" id="create secrets table" 2024-03-11T20:52:56.499 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.142509051Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-11T20:52:56.499 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.14679181Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-11T20:52:56.499 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.147684757Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-11T20:52:56.499 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.147967269Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-11T20:52:56.499 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.152218143Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-11T20:52:56.499 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.156466649Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-11T20:52:56.499 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.156833189Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-11T20:52:56.499 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.157245915Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-11T20:52:56.499 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.157557366Z level=info msg="Executing migration" id="create permission table" 2024-03-11T20:52:56.499 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.15790872Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-11T20:52:56.499 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.158320407Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-11T20:52:56.499 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.158722006Z level=info msg="Executing migration" id="create role table" 2024-03-11T20:52:56.500 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.159089005Z level=info msg="Executing migration" id="add column display_name" 2024-03-11T20:52:56.500 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.160017164Z level=info msg="Executing migration" id="add column group_name" 2024-03-11T20:52:56.501 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.160898627Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-11T20:52:56.501 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.161300387Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-11T20:52:56.501 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.161714717Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-11T20:52:56.501 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.162127452Z level=info msg="Executing migration" id="create team role table" 2024-03-11T20:52:56.501 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.162486145Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-11T20:52:56.501 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.162915802Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-11T20:52:56.501 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.163348199Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-11T20:52:56.501 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.163738243Z level=info msg="Executing migration" id="create user role table" 2024-03-11T20:52:56.501 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.164103402Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-11T20:52:56.501 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.164498051Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-11T20:52:56.501 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.164887075Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-11T20:52:56.501 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.165289369Z level=info msg="Executing migration" id="create builtin role table" 2024-03-11T20:52:56.501 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.165670313Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-11T20:52:56.502 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.166074787Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-11T20:52:56.502 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.166465558Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-11T20:52:56.502 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.167427794Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-11T20:52:56.502 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.167865692Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-11T20:52:56.502 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.16827734Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-11T20:52:56.502 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.168657681Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-11T20:52:56.502 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.16902844Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-11T20:52:56.502 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.16937993Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-11T20:52:56.502 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.16980096Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-11T20:52:56.502 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.170729189Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-11T20:52:56.502 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.171147732Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-11T20:52:56.502 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.17158507Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-11T20:52:56.502 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.171846529Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-11T20:52:56.502 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.172245352Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-11T20:52:56.503 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.172686094Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-11T20:52:56.503 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.173012862Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-11T20:52:56.503 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.173301177Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-11T20:52:56.503 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.173611095Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-11T20:52:56.503 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.173978071Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-11T20:52:56.503 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.174445531Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-11T20:52:56.503 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.175466034Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-11T20:52:56.503 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.175760693Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-11T20:52:56.503 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.176159023Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-11T20:52:56.503 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.176575877Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-11T20:52:56.503 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.17700146Z level=info msg="Executing migration" id="add correlation config column" 2024-03-11T20:52:56.503 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.177984506Z level=info msg="Executing migration" id="create entity_events table" 2024-03-11T20:52:56.503 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.178386117Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-11T20:52:56.503 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.178766449Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T20:52:56.504 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.17885692Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T20:52:56.504 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.179133585Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T20:52:56.504 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.179224921Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T20:52:56.504 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.179469087Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-11T20:52:56.504 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.179826033Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-11T20:52:56.504 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.180264106Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-11T20:52:56.504 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.180707578Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T20:52:56.504 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.181166321Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-11T20:52:56.504 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.181587082Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T20:52:56.504 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.18198001Z level=info msg="Executing migration" id="Drop public config table" 2024-03-11T20:52:56.504 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.182401279Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-11T20:52:56.504 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.182770864Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-11T20:52:56.504 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.183205729Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T20:52:56.504 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.183668848Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-11T20:52:56.505 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.184130192Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-11T20:52:56.505 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.187122334Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-11T20:52:56.505 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.188127648Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-11T20:52:56.505 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.189099499Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-11T20:52:56.505 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.189389111Z level=info msg="Executing migration" id="add share column" 2024-03-11T20:52:56.505 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.190427584Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-11T20:52:56.505 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.191076368Z level=info msg="Executing migration" id="create file table" 2024-03-11T20:52:56.505 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.191508266Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-11T20:52:56.505 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.191951803Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-11T20:52:56.505 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.192461494Z level=info msg="Executing migration" id="create file_meta table" 2024-03-11T20:52:56.505 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.192851911Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-11T20:52:56.505 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.193354473Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-11T20:52:56.505 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.193649924Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-11T20:52:56.505 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.194006002Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-11T20:52:56.506 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.194323439Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-11T20:52:56.506 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.194709337Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-11T20:52:56.506 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.195760227Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-11T20:52:56.506 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.196089171Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-11T20:52:56.506 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.19653361Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-11T20:52:56.506 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.196878529Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-11T20:52:56.506 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.197179485Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-11T20:52:56.506 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.197547634Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-11T20:52:56.506 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.198568706Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-11T20:52:56.506 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.199575999Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-11T20:52:56.506 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.200027864Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-11T20:52:56.506 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.209875235Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-11T20:52:56.506 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.210338353Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-11T20:52:56.506 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.210780882Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-11T20:52:56.507 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.214294276Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-11T20:52:56.507 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.214616596Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-11T20:52:56.507 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.214901815Z level=info msg="Executing migration" id="create folder table" 2024-03-11T20:52:56.507 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.215285356Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-11T20:52:56.507 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.215730602Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-11T20:52:56.507 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.216190872Z level=info msg="Executing migration" id="Update folder title length" 2024-03-11T20:52:56.507 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.216479738Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-11T20:52:56.507 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-11T20:52:56.216901901Z level=info msg="migrations completed" performed=485 skipped=0 duration=249.293547ms 2024-03-11T20:52:56.507 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=sqlstore t=2024-03-11T20:52:56.217299883Z level=info msg="Created default organization" 2024-03-11T20:52:56.507 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=plugin.loader t=2024-03-11T20:52:56.272743528Z level=info msg="Plugin registered" pluginID=input 2024-03-11T20:52:56.507 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=plugin.loader t=2024-03-11T20:52:56.300653178Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-11T20:52:56.507 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=plugin.loader t=2024-03-11T20:52:56.300675719Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-11T20:52:56.507 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=secrets t=2024-03-11T20:52:56.300726354Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-11T20:52:56.507 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=query_data t=2024-03-11T20:52:56.302830374Z level=info msg="Query Service initialization" 2024-03-11T20:52:56.508 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=live.push_http t=2024-03-11T20:52:56.305267666Z level=info msg="Live Push Gateway initialization" 2024-03-11T20:52:57.229 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:52:57 smithi129 bash[80208]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-11T20:52:57.236 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=infra.usagestats.collector t=2024-03-11T20:52:56.923504383Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-11T20:52:57.236 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=provisioning.datasources t=2024-03-11T20:52:56.924454763Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-11T20:52:57.236 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=provisioning.datasources t=2024-03-11T20:52:56.935273405Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-11T20:52:57.236 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=provisioning.alerting t=2024-03-11T20:52:56.945831793Z level=info msg="starting to provision alerting" 2024-03-11T20:52:57.236 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=provisioning.alerting t=2024-03-11T20:52:56.945855025Z level=info msg="finished to provision alerting" 2024-03-11T20:52:57.236 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=ngalert.state.manager t=2024-03-11T20:52:56.946065212Z level=info msg="Warming state cache for startup" 2024-03-11T20:52:57.236 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=grafanaStorageLogger t=2024-03-11T20:52:56.946094876Z level=info msg="storage starting" 2024-03-11T20:52:57.237 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=ngalert.state.manager t=2024-03-11T20:52:56.946180367Z level=info msg="State cache has been initialized" states=0 duration=114.042µs 2024-03-11T20:52:57.237 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=ticker t=2024-03-11T20:52:56.946246698Z level=info msg=starting first_tick=2024-03-11T20:53:00Z 2024-03-11T20:52:57.237 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=ngalert.multiorg.alertmanager t=2024-03-11T20:52:56.946272167Z level=info msg="starting MultiOrg Alertmanager" 2024-03-11T20:52:57.237 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=http.server t=2024-03-11T20:52:56.947887982Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-11T20:52:57.237 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 20:52:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=sqlstore.transactions t=2024-03-11T20:52:56.966126733Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T20:52:57.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:57 smithi129 ceph-mon[30184]: Deploying daemon node-exporter.a on smithi129 2024-03-11T20:52:57.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:57 smithi129 ceph-mon[42066]: Deploying daemon node-exporter.a on smithi129 2024-03-11T20:52:57.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:57 smithi157 ceph-mon[33359]: Deploying daemon node-exporter.a on smithi129 2024-03-11T20:52:58.228 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:52:57 smithi129 bash[80208]: Getting image source signatures 2024-03-11T20:52:58.228 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:52:57 smithi129 bash[80208]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-11T20:52:58.228 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:52:57 smithi129 bash[80208]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-11T20:52:58.229 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:52:57 smithi129 bash[80208]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-11T20:52:58.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:52:58 smithi129 ceph-mon[30184]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:58.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:52:58 smithi129 ceph-mon[42066]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:52:58.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:52:58 smithi157 ceph-mon[33359]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:00.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:00 smithi129 ceph-mon[30184]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:00.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:00 smithi129 ceph-mon[42066]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:00.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:00 smithi157 ceph-mon[33359]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:01.228 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:00 smithi129 bash[80208]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-11T20:53:01.228 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 bash[80208]: Writing manifest to image destination 2024-03-11T20:53:01.702 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 podman[80208]: 2024-03-11 20:53:01.348160904 +0000 UTC m=+4.322193174 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T20:53:01.702 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 podman[80208]: 2024-03-11 20:53:01.463502109 +0000 UTC m=+4.437534423 container create 92d577d3bf30dfa5034d7b23fab2a52fcb0a3a46ee06f5c7173137cbd1d6e5e0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T20:53:01.703 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 podman[80208]: 2024-03-11 20:53:01.707278334 +0000 UTC m=+4.681310611 container init 92d577d3bf30dfa5034d7b23fab2a52fcb0a3a46ee06f5c7173137cbd1d6e5e0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T20:53:01.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-mon[30184]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:01.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:01.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:01.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-mon[42066]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:01.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:01.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:01.979 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 podman[80208]: 2024-03-11 20:53:01.712815109 +0000 UTC m=+4.686847377 container start 92d577d3bf30dfa5034d7b23fab2a52fcb0a3a46ee06f5c7173137cbd1d6e5e0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T20:53:01.979 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.716Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T20:53:01.980 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.716Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T20:53:01.980 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.717Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T20:53:01.980 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.717Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T20:53:01.980 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.717Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T20:53:01.980 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.717Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T20:53:01.980 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T20:53:01.980 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T20:53:01.980 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T20:53:01.980 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T20:53:01.980 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T20:53:01.980 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T20:53:01.980 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T20:53:01.980 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T20:53:01.981 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T20:53:01.981 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T20:53:01.981 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T20:53:01.981 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T20:53:01.981 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T20:53:01.981 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T20:53:01.981 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T20:53:01.981 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T20:53:01.981 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T20:53:01.981 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T20:53:01.981 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T20:53:01.981 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T20:53:01.981 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T20:53:01.981 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T20:53:01.981 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T20:53:01.982 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T20:53:01.982 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T20:53:01.982 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T20:53:01.982 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T20:53:01.982 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T20:53:01.982 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T20:53:01.982 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T20:53:01.982 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T20:53:01.982 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T20:53:01.982 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T20:53:01.983 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T20:53:01.983 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T20:53:01.983 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T20:53:01.983 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T20:53:01.983 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T20:53:01.983 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T20:53:01.983 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T20:53:01.983 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T20:53:01.983 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T20:53:01.983 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T20:53:01.983 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T20:53:01.983 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T20:53:01.983 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T20:53:01.983 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a[80256]: ts=2024-03-11T20:53:01.718Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T20:53:01.984 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 bash[80208]: 92d577d3bf30dfa5034d7b23fab2a52fcb0a3a46ee06f5c7173137cbd1d6e5e0 2024-03-11T20:53:01.984 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 20:53:01 smithi129 systemd[1]: Started Ceph node-exporter.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T20:53:02.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:01 smithi157 ceph-mon[33359]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:02.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:02.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:03.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:02 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:03.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:02 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:03.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:02 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:03.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:02 smithi129 ceph-mon[42066]: Deploying daemon node-exporter.b on smithi157 2024-03-11T20:53:03.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:02 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:03.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:02 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:03.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:02 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:03.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:02 smithi129 ceph-mon[30184]: Deploying daemon node-exporter.b on smithi157 2024-03-11T20:53:03.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:02 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:03.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:02 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:03.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:02 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:03.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:02 smithi157 ceph-mon[33359]: Deploying daemon node-exporter.b on smithi157 2024-03-11T20:53:03.236 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:03 smithi157 bash[66561]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-11T20:53:03.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:53:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:53:04.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:03 smithi129 ceph-mon[30184]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:04.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:03 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:04.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:03 smithi129 ceph-mon[42066]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:04.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:03 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:04.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:03 smithi157 ceph-mon[33359]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:04.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:03 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:04.236 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:03 smithi157 bash[66561]: Getting image source signatures 2024-03-11T20:53:04.236 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:03 smithi157 bash[66561]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-11T20:53:04.236 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:03 smithi157 bash[66561]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-11T20:53:04.236 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:03 smithi157 bash[66561]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-11T20:53:05.486 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 bash[66561]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-11T20:53:05.486 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 bash[66561]: Writing manifest to image destination 2024-03-11T20:53:05.760 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 podman[66561]: 2024-03-11 20:53:05.51069742 +0000 UTC m=+2.515209173 container create c176bd34aee40458fbb254a777eb13107cebcedf914765a1ae427bb58a22f543 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T20:53:05.760 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 podman[66561]: 2024-03-11 20:53:05.429031347 +0000 UTC m=+2.433543119 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T20:53:05.760 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 podman[66561]: 2024-03-11 20:53:05.754452823 +0000 UTC m=+2.758964581 container init c176bd34aee40458fbb254a777eb13107cebcedf914765a1ae427bb58a22f543 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T20:53:05.760 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 podman[66561]: 2024-03-11 20:53:05.76049122 +0000 UTC m=+2.765002976 container start c176bd34aee40458fbb254a777eb13107cebcedf914765a1ae427bb58a22f543 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T20:53:06.228 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:53:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:53:06.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:05 smithi129 ceph-mon[30184]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:06.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:06.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:06.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:05 smithi129 ceph-mon[42066]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:06.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:06.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:06.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-mon[33359]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:06.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:06.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:06.236 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.764Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T20:53:06.237 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.764Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T20:53:06.237 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.765Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T20:53:06.237 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.765Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T20:53:06.237 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.765Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T20:53:06.237 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.765Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T20:53:06.237 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T20:53:06.237 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T20:53:06.237 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T20:53:06.237 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T20:53:06.237 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T20:53:06.237 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T20:53:06.237 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T20:53:06.238 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T20:53:06.238 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T20:53:06.238 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T20:53:06.238 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T20:53:06.238 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T20:53:06.238 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T20:53:06.238 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T20:53:06.238 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T20:53:06.238 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T20:53:06.238 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T20:53:06.238 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T20:53:06.238 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T20:53:06.238 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T20:53:06.239 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T20:53:06.239 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T20:53:06.239 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T20:53:06.239 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T20:53:06.239 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T20:53:06.239 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T20:53:06.239 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T20:53:06.239 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T20:53:06.239 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T20:53:06.239 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T20:53:06.239 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T20:53:06.239 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T20:53:06.239 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T20:53:06.239 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T20:53:06.240 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T20:53:06.240 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T20:53:06.240 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T20:53:06.240 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T20:53:06.240 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T20:53:06.240 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T20:53:06.240 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T20:53:06.240 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T20:53:06.240 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T20:53:06.240 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T20:53:06.240 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T20:53:06.240 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T20:53:06.240 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b[66612]: ts=2024-03-11T20:53:05.766Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T20:53:06.240 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 bash[66561]: c176bd34aee40458fbb254a777eb13107cebcedf914765a1ae427bb58a22f543 2024-03-11T20:53:06.241 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 20:53:05 smithi157 systemd[1]: Started Ceph node-exporter.b for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T20:53:07.183 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:06 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:07.183 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:06 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:07.183 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:06 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:07.183 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:06 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:53:07.183 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:06 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:07.183 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:06 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:07.183 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:06 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:07.183 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:06 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:53:07.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:06 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:07.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:06 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:07.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:06 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:07.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:06 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:53:08.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:07 smithi129 ceph-mon[30184]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:07 smithi129 ceph-mon[42066]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:08.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:07 smithi157 ceph-mon[33359]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:10.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:10 smithi129 ceph-mon[30184]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:10.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:10 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:10.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:10 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:10.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:10 smithi129 ceph-mon[42066]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:10.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:10 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:10.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:10 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:10.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:10 smithi157 ceph-mon[33359]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:10.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:10 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:10.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:10 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:11.709 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:11 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:11.709 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:11 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:11.709 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:11 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:11.709 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:11 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:53:11.709 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:11 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:53:11.709 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:11 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:11.709 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:11 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:11.709 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:11 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:11.710 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:11 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:11.710 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:11 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:53:11.710 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:11 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:53:11.710 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:11 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:11.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:11 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:11.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:11 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:11.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:11 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:11.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:11 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:53:11.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:11 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:53:11.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:11 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:12.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:12 smithi129 ceph-mon[30184]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:12.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:12 smithi129 ceph-mon[30184]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T20:53:12.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:12 smithi129 ceph-mon[30184]: Reconfiguring daemon alertmanager.a on smithi129 2024-03-11T20:53:12.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:12 smithi129 ceph-mon[42066]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:12.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:12 smithi129 ceph-mon[42066]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T20:53:12.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:12 smithi129 ceph-mon[42066]: Reconfiguring daemon alertmanager.a on smithi129 2024-03-11T20:53:12.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:12 smithi157 ceph-mon[33359]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:12.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:12 smithi157 ceph-mon[33359]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T20:53:12.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:12 smithi157 ceph-mon[33359]: Reconfiguring daemon alertmanager.a on smithi129 2024-03-11T20:53:13.735 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:53:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:53:14.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:14 smithi129 ceph-mon[30184]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:14.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:14 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:14.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:14 smithi129 ceph-mon[42066]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:14.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:14 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:14.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:14 smithi157 ceph-mon[33359]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:14.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:14 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:15.200 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:14 smithi129 systemd[1]: Stopping Ceph alertmanager.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T20:53:15.478 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:15 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[79919]: ts=2024-03-11T20:53:15.204Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T20:53:15.478 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:15 smithi129 podman[81072]: 2024-03-11 20:53:15.225373494 +0000 UTC m=+0.150576997 container died 6ec5f38dae02cd027a42c34df1ca29799a9b332a755d8e9d9bec9a87ba6191ee (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T20:53:15.978 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:15 smithi129 podman[81072]: 2024-03-11 20:53:15.72848485 +0000 UTC m=+0.653688349 container cleanup 6ec5f38dae02cd027a42c34df1ca29799a9b332a755d8e9d9bec9a87ba6191ee (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T20:53:15.978 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:15 smithi129 bash[81072]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T20:53:16.244 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:53:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:53:16.244 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:15 smithi129 podman[81091]: 2024-03-11 20:53:15.996762405 +0000 UTC m=+0.767723744 container remove 6ec5f38dae02cd027a42c34df1ca29799a9b332a755d8e9d9bec9a87ba6191ee (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T20:53:16.244 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 podman[81091]: 2024-03-11 20:53:16.097508728 +0000 UTC m=+0.868470061 volume remove ed93668f4cd9ef0bff0559f85bb97d96ecbe7d689269db5f2eed4f6e27914425 2024-03-11T20:53:16.594 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:16 smithi129 ceph-mon[30184]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:16.594 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:16 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:16.595 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:16 smithi129 ceph-mon[42066]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:16.595 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:16 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:16.595 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T20:53:16.595 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 systemd[1]: Stopped Ceph alertmanager.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T20:53:16.595 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 systemd[1]: Starting Ceph alertmanager.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T20:53:16.595 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 podman[81190]: 2024-03-11 20:53:16.38078628 +0000 UTC m=+0.019783562 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-11T20:53:16.595 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 podman[81190]: 2024-03-11 20:53:16.490320807 +0000 UTC m=+0.129318074 volume create a106844fbf46eec90b3993bf6fca813d3c63da11b7eac7235dc39ef82356a0be 2024-03-11T20:53:16.595 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 podman[81190]: 2024-03-11 20:53:16.599201287 +0000 UTC m=+0.238198545 container create 92cc4ee6779d25ec680f9ab23c18d098ce7553fef5e6d82a2de7a7ee90759b8f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T20:53:16.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:16 smithi157 ceph-mon[33359]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:16.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:16 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:17.228 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 podman[81190]: 2024-03-11 20:53:16.927641489 +0000 UTC m=+0.566638752 container init 92cc4ee6779d25ec680f9ab23c18d098ce7553fef5e6d82a2de7a7ee90759b8f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T20:53:17.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 podman[81190]: 2024-03-11 20:53:16.933158551 +0000 UTC m=+0.572155810 container start 92cc4ee6779d25ec680f9ab23c18d098ce7553fef5e6d82a2de7a7ee90759b8f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T20:53:17.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[81204]: ts=2024-03-11T20:53:16.945Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-11T20:53:17.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[81204]: ts=2024-03-11T20:53:16.945Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-11T20:53:17.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[81204]: ts=2024-03-11T20:53:16.946Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.129 port=9094 2024-03-11T20:53:17.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[81204]: ts=2024-03-11T20:53:16.947Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T20:53:17.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[81204]: ts=2024-03-11T20:53:16.974Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T20:53:17.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[81204]: ts=2024-03-11T20:53:16.974Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T20:53:17.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[81204]: ts=2024-03-11T20:53:16.977Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-11T20:53:17.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[81204]: ts=2024-03-11T20:53:16.977Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-11T20:53:17.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:17 smithi129 bash[81190]: 92cc4ee6779d25ec680f9ab23c18d098ce7553fef5e6d82a2de7a7ee90759b8f 2024-03-11T20:53:17.229 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:17 smithi129 systemd[1]: Started Ceph alertmanager.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T20:53:18.213 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:18 smithi157 ceph-mon[33359]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:18.213 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:18 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:18.214 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:18 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:18.214 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:18 smithi157 ceph-mon[33359]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T20:53:18.214 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:18 smithi157 ceph-mon[33359]: Reconfiguring daemon prometheus.a on smithi157 2024-03-11T20:53:18.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:18 smithi129 ceph-mon[30184]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:18.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:18 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:18.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:18 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:18.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:18 smithi129 ceph-mon[30184]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T20:53:18.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:18 smithi129 ceph-mon[30184]: Reconfiguring daemon prometheus.a on smithi157 2024-03-11T20:53:18.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:18 smithi129 ceph-mon[42066]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:18.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:18 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:18.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:18 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:18.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:18 smithi129 ceph-mon[42066]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T20:53:18.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:18 smithi129 ceph-mon[42066]: Reconfiguring daemon prometheus.a on smithi157 2024-03-11T20:53:19.228 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:18 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[81204]: ts=2024-03-11T20:53:18.948Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00070364s 2024-03-11T20:53:20.468 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:20 smithi157 ceph-mon[33359]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:20.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:20 smithi129 ceph-mon[30184]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:20.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:20 smithi129 ceph-mon[42066]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:20.736 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:20 smithi157 systemd[1]: Stopping Ceph prometheus.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T20:53:21.156 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:53:20.887Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T20:53:21.157 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:53:20.888Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T20:53:21.157 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:53:20.888Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T20:53:21.157 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:53:20.888Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T20:53:21.157 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:53:20.888Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T20:53:21.157 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:53:20.888Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T20:53:21.157 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:53:20.888Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T20:53:21.157 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:53:20.888Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T20:53:21.157 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:53:20.888Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T20:53:21.157 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:53:20.888Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T20:53:21.157 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:53:20.888Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T20:53:21.157 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[64287]: ts=2024-03-11T20:53:20.888Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T20:53:21.157 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:20 smithi157 podman[67660]: 2024-03-11 20:53:20.898286399 +0000 UTC m=+0.128187150 container died 3f2859b29719b4748ac770b38078c33445c856e2060b658629b053137882ab0e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T20:53:21.486 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:21 smithi157 podman[67660]: 2024-03-11 20:53:21.268119555 +0000 UTC m=+0.498020296 container cleanup 3f2859b29719b4748ac770b38078c33445c856e2060b658629b053137882ab0e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T20:53:21.486 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:21 smithi157 bash[67660]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T20:53:22.323 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:22 smithi157 ceph-mon[33359]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:22.324 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:22 smithi157 podman[67678]: 2024-03-11 20:53:22.203070312 +0000 UTC m=+1.306730244 container remove 3f2859b29719b4748ac770b38078c33445c856e2060b658629b053137882ab0e (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T20:53:22.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:22 smithi129 ceph-mon[30184]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:22.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:22 smithi129 ceph-mon[42066]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:22.736 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:22 smithi157 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-11T20:53:22.736 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:22 smithi157 systemd[1]: Stopped Ceph prometheus.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T20:53:22.736 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:22 smithi157 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@prometheus.a.service: Consumed 1.174s CPU time. 2024-03-11T20:53:22.736 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:22 smithi157 systemd[1]: Starting Ceph prometheus.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T20:53:22.736 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:22 smithi157 podman[67779]: 2024-03-11 20:53:22.471741934 +0000 UTC m=+0.019624024 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-11T20:53:23.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:53:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:53:23.986 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:23 smithi157 podman[67779]: 2024-03-11 20:53:23.627365227 +0000 UTC m=+1.175247316 container create 5c244fcc3377878e6859f40deb86710bb84bccd4f52fa1b843be1fc981ab632d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T20:53:24.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:24 smithi129 ceph-mon[30184]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:24.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:24 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:24 smithi129 ceph-mon[42066]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:24 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:24.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:24 smithi157 ceph-mon[33359]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:24 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:25.987 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 podman[67779]: 2024-03-11 20:53:25.665904796 +0000 UTC m=+3.213786889 container init 5c244fcc3377878e6859f40deb86710bb84bccd4f52fa1b843be1fc981ab632d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T20:53:25.987 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 podman[67779]: 2024-03-11 20:53:25.672968247 +0000 UTC m=+3.220850344 container start 5c244fcc3377878e6859f40deb86710bb84bccd4f52fa1b843be1fc981ab632d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T20:53:25.987 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.700Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T20:53:25.987 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.701Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T20:53:25.987 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.701Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi157 (none))" 2024-03-11T20:53:25.987 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.701Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T20:53:25.987 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.701Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T20:53:25.988 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.702Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T20:53:25.988 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.703Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T20:53:25.988 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.705Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T20:53:25.988 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.705Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T20:53:25.988 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.706Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T20:53:25.988 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.706Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.477µs 2024-03-11T20:53:25.988 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.706Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T20:53:25.988 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.707Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-11T20:53:25.988 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.707Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-11T20:53:25.988 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.707Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=33.818µs wal_replay_duration=551.12µs wbl_replay_duration=138ns total_replay_duration=615.257µs 2024-03-11T20:53:25.988 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.708Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T20:53:25.988 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.708Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T20:53:25.988 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.708Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T20:53:25.989 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.729Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.165062ms db_storage=977ns remote_storage=1.51µs web_handler=414ns query_engine=673ns scrape=574.054µs scrape_sd=128.034µs notify=15.347µs notify_sd=17.065µs rules=19.982983ms tracing=7.456µs 2024-03-11T20:53:25.989 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.729Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T20:53:25.989 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T20:53:25.729Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T20:53:26.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-mon[30184]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:26.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-mon[42066]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:26.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:26 smithi157 ceph-mon[33359]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:26.978 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[81204]: ts=2024-03-11T20:53:26.951Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003997802s 2024-03-11T20:53:26.979 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:26] ENGINE Bus STOPPING 2024-03-11T20:53:26.979 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T20:53:26.979 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:26] ENGINE Bus STOPPED 2024-03-11T20:53:26.979 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:26] ENGINE Bus STARTING 2024-03-11T20:53:26.979 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:26] ENGINE Serving on http://:::9283 2024-03-11T20:53:26.979 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:26] ENGINE Bus STARTED 2024-03-11T20:53:26.979 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:26] ENGINE Bus STOPPING 2024-03-11T20:53:26.979 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T20:53:26.979 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:26] ENGINE Bus STOPPED 2024-03-11T20:53:26.979 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:26] ENGINE Bus STARTING 2024-03-11T20:53:26.979 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:26] ENGINE Serving on http://:::9283 2024-03-11T20:53:26.980 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:26] ENGINE Bus STARTED 2024-03-11T20:53:26.980 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:26] ENGINE Bus STOPPING 2024-03-11T20:53:26.980 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T20:53:26.980 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:26] ENGINE Bus STOPPED 2024-03-11T20:53:26.980 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:26] ENGINE Bus STARTING 2024-03-11T20:53:26.985 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:26 smithi157 bash[67779]: 5c244fcc3377878e6859f40deb86710bb84bccd4f52fa1b843be1fc981ab632d 2024-03-11T20:53:26.985 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 20:53:26 smithi157 systemd[1]: Started Ceph prometheus.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T20:53:27.478 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:27] ENGINE Serving on http://:::9283 2024-03-11T20:53:27.478 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:20:53:27] ENGINE Bus STARTED 2024-03-11T20:53:27.901 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:27.901 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:27.901 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T20:53:27.901 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi129.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T20:53:27.901 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:27.901 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T20:53:27.901 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi157.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T20:53:27.902 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:27.902 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T20:53:27.902 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi157.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T20:53:27.902 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:27.902 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:53:27.902 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:27.902 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:27.902 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T20:53:27.902 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi129.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T20:53:27.902 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:27.903 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T20:53:27.903 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi157.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T20:53:27.903 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:27.903 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T20:53:27.903 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi157.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T20:53:27.903 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:27.903 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:27 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:53:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:27 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:27 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:27 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T20:53:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:27 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi129.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T20:53:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:27 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:27 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T20:53:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:27 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi157.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T20:53:27.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:27 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:27.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:27 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T20:53:27.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:27 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi157.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T20:53:27.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:27 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:27.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:27 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:53:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:28 smithi129 ceph-mon[30184]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T20:53:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:28 smithi129 ceph-mon[30184]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi129.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T20:53:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:28 smithi129 ceph-mon[30184]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T20:53:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:28 smithi129 ceph-mon[30184]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi157.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T20:53:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:28 smithi129 ceph-mon[30184]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T20:53:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:28 smithi129 ceph-mon[30184]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi157.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T20:53:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:28 smithi129 ceph-mon[30184]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:28.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:28 smithi129 ceph-mon[42066]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T20:53:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:28 smithi129 ceph-mon[42066]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi129.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T20:53:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:28 smithi129 ceph-mon[42066]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T20:53:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:28 smithi129 ceph-mon[42066]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi157.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T20:53:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:28 smithi129 ceph-mon[42066]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T20:53:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:28 smithi129 ceph-mon[42066]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi157.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T20:53:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:28 smithi129 ceph-mon[42066]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:28 smithi157 ceph-mon[33359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T20:53:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:28 smithi157 ceph-mon[33359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi129.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T20:53:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:28 smithi157 ceph-mon[33359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T20:53:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:28 smithi157 ceph-mon[33359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi157.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T20:53:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:28 smithi157 ceph-mon[33359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T20:53:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:28 smithi157 ceph-mon[33359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi157.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T20:53:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:28 smithi157 ceph-mon[33359]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:30.961 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:30 smithi157 ceph-mon[33359]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:30.961 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:30 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:30.961 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:30 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:30.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:30 smithi129 ceph-mon[30184]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:30.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:30 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:30.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:30 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:30.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:30 smithi129 ceph-mon[42066]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:30.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:30 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:30.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:30 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:31.855 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:31 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:31.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:31 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:31.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:31 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:32.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:32 smithi157 ceph-mon[33359]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:32.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:32 smithi129 ceph-mon[30184]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:32.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:32 smithi129 ceph-mon[42066]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:33.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:53:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:53:34.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:33 smithi129 ceph-mon[30184]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:34.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:33 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:34.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:33 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:34.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:33 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:53:34.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:33 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:53:34.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:33 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:34.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:33 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:53:34.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:33 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:34.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:33 smithi129 ceph-mon[42066]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:34.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:33 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:34.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:33 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:34.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:33 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:53:34.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:33 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:53:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:33 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:33 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:53:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:33 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:33 smithi157 ceph-mon[33359]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:33 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:33 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:33 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:53:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:33 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:53:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:33 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:33 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:53:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:33 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:36.478 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:53:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:53:36.872 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:36 smithi129 ceph-mon[30184]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:36.873 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:36 smithi129 ceph-mon[42066]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:36.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:36 smithi157 ceph-mon[33359]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:38.408 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:38 smithi129 ceph-mon[42066]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:38.409 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:38 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:38.409 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:38 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:38.409 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:38 smithi129 ceph-mon[30184]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:38.409 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:38 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:38.409 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:38 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:38.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:38 smithi157 ceph-mon[33359]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:38.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:38 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:38.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:38 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:40.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:40 smithi129 ceph-mon[30184]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:40 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:40 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:40 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:53:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:40 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:53:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:40 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:40.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:40 smithi129 ceph-mon[42066]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:40.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:40 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:40 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:40 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:53:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:40 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:53:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:40 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:40 smithi157 ceph-mon[33359]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:40 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:40 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:40 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:53:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:40 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:53:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:40 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:53:42.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:42 smithi129 ceph-mon[30184]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:42.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:42 smithi129 ceph-mon[42066]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:42.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:42 smithi157 ceph-mon[33359]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:43.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:53:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:53:44.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:44 smithi129 ceph-mon[30184]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:44.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:44 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:44.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:44 smithi129 ceph-mon[42066]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:44.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:44 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:44.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:44 smithi157 ceph-mon[33359]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:44.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:44 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:46.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:46 smithi129 ceph-mon[30184]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:46.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:46 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:46.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:53:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:53:46.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:46 smithi129 ceph-mon[42066]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:46.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:46 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:46.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:46 smithi157 ceph-mon[33359]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:46.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:46 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:53:48.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:48 smithi129 ceph-mon[30184]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:48.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:48 smithi129 ceph-mon[42066]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:48.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:48 smithi157 ceph-mon[33359]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:50.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:50 smithi129 ceph-mon[30184]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:50.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:50 smithi129 ceph-mon[42066]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:50.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:50 smithi157 ceph-mon[33359]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:52.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:52 smithi129 ceph-mon[30184]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:52.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:52 smithi129 ceph-mon[42066]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:52.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:52 smithi157 ceph-mon[33359]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:53:52.547 INFO:tasks.workunit.client.0.smithi129.stderr:Note: switching to '1c349451176cc5b4ebfb24b22eaaa754e05cff6c'. 2024-03-11T20:53:52.547 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-03-11T20:53:52.547 INFO:tasks.workunit.client.0.smithi129.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-11T20:53:52.547 INFO:tasks.workunit.client.0.smithi129.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-11T20:53:52.547 INFO:tasks.workunit.client.0.smithi129.stderr:state without impacting any branches by switching back to a branch. 2024-03-11T20:53:52.547 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-03-11T20:53:52.547 INFO:tasks.workunit.client.0.smithi129.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-11T20:53:52.547 INFO:tasks.workunit.client.0.smithi129.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-11T20:53:52.547 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-03-11T20:53:52.547 INFO:tasks.workunit.client.0.smithi129.stderr: git switch -c 2024-03-11T20:53:52.548 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-03-11T20:53:52.548 INFO:tasks.workunit.client.0.smithi129.stderr:Or undo this operation with: 2024-03-11T20:53:52.548 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-03-11T20:53:52.548 INFO:tasks.workunit.client.0.smithi129.stderr: git switch - 2024-03-11T20:53:52.548 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-03-11T20:53:52.548 INFO:tasks.workunit.client.0.smithi129.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-11T20:53:52.548 INFO:tasks.workunit.client.0.smithi129.stderr: 2024-03-11T20:53:52.548 INFO:tasks.workunit.client.0.smithi129.stderr:HEAD is now at 1c349451176 Merge pull request #56077 from idryomov/wip-rbd-mirror-bash 2024-03-11T20:53:52.549 DEBUG:teuthology.orchestra.run.smithi129:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-11T20:53:52.579 INFO:tasks.workunit.client.0.smithi129.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-11T20:53:52.581 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-11T20:53:52.582 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-11T20:53:53.335 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-11T20:53:53.417 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-11T20:53:53.456 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-11T20:53:53.457 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-11T20:53:53.457 INFO:tasks.workunit.client.0.smithi129.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-11T20:53:53.496 INFO:tasks.workunit.client.0.smithi129.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-11T20:53:53.499 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-03-11T20:53:53.499 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-11T20:53:53.555 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-03-11T20:53:53.556 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-03-11T20:53:53.557 DEBUG:teuthology.orchestra.run.smithi129:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-03-11T20:53:53.557 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:53:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:53:53.624 INFO:tasks.workunit.client.0.smithi129.stderr:+ parallel=1 2024-03-11T20:53:53.624 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' '' = --serial ']' 2024-03-11T20:53:53.624 INFO:tasks.workunit.client.0.smithi129.stderr:+ crimson=0 2024-03-11T20:53:53.624 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' '' = --crimson ']' 2024-03-11T20:53:53.624 INFO:tasks.workunit.client.0.smithi129.stderr:+ color= 2024-03-11T20:53:53.624 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -t 1 ']' 2024-03-11T20:53:53.624 INFO:tasks.workunit.client.0.smithi129.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-03-11T20:53:53.624 INFO:tasks.workunit.client.0.smithi129.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-11T20:53:53.624 INFO:tasks.workunit.client.0.smithi129.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-11T20:53:53.625 INFO:tasks.workunit.client.0.smithi129.stderr:+ declare -A pids 2024-03-11T20:53:53.625 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.626 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_aio 2024-03-11T20:53:53.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_aio' 2024-03-11T20:53:53.626 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_aio 2024-03-11T20:53:53.626 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_aio 2024-03-11T20:53:53.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82481 2024-03-11T20:53:53.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_aio on pid 82481' 2024-03-11T20:53:53.627 INFO:tasks.workunit.client.0.smithi129.stdout:test api_aio on pid 82481 2024-03-11T20:53:53.630 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82481 2024-03-11T20:53:53.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-03-11T20:53:53.631 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_aio_pp 2024-03-11T20:53:53.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_aio_pp' 2024-03-11T20:53:53.631 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_aio_pp 2024-03-11T20:53:53.631 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.631 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.631 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.631 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.632 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.632 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.632 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.632 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.632 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.632 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.632 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.632 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.632 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.632 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.632 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.632 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.632 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.633 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.633 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.633 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.633 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.633 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.633 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.633 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.633 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_aio_pp 2024-03-11T20:53:53.633 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82488 2024-03-11T20:53:53.633 INFO:tasks.workunit.client.0.smithi129.stdout:test api_aio_pp on pid 82488 2024-03-11T20:53:53.638 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_aio_pp on pid 82488' 2024-03-11T20:53:53.639 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82488 2024-03-11T20:53:53.639 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.639 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.639 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-03-11T20:53:53.639 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_io 2024-03-11T20:53:53.639 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_io' 2024-03-11T20:53:53.639 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_io 2024-03-11T20:53:53.639 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.640 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.640 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.640 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.640 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.640 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.640 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.640 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.640 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.640 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.640 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.640 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.641 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.641 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.641 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.642 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.642 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.642 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.642 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.642 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.642 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.642 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.642 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.643 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.643 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.643 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.643 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.643 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.643 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.643 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.643 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.643 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.643 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.643 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.643 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.643 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.644 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.644 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.644 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_io 2024-03-11T20:53:53.644 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82497 2024-03-11T20:53:53.644 INFO:tasks.workunit.client.0.smithi129.stdout:test api_io on pid 82497 2024-03-11T20:53:53.650 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_io on pid 82497' 2024-03-11T20:53:53.650 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82497 2024-03-11T20:53:53.650 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-03-11T20:53:53.650 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.650 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.650 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_io_pp 2024-03-11T20:53:53.650 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_io_pp' 2024-03-11T20:53:53.651 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_io_pp 2024-03-11T20:53:53.651 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.651 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.651 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.651 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.651 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.651 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.651 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.652 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.652 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.652 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.652 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.652 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.652 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.652 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.652 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.652 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.652 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.652 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.652 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.652 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.652 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.653 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.653 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.653 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.653 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.653 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.653 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.653 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.653 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.653 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.653 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.653 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.653 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.653 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.654 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.654 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.654 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.654 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.655 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.655 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.655 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.655 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.655 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.655 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.655 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.655 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_io_pp 2024-03-11T20:53:53.655 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82508 2024-03-11T20:53:53.655 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_io_pp on pid 82508' 2024-03-11T20:53:53.656 INFO:tasks.workunit.client.0.smithi129.stdout:test api_io_pp on pid 82508 2024-03-11T20:53:53.661 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82508 2024-03-11T20:53:53.661 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.661 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-03-11T20:53:53.661 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.661 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_asio 2024-03-11T20:53:53.661 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_asio' 2024-03-11T20:53:53.662 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_asio 2024-03-11T20:53:53.662 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.662 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.662 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.662 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.662 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.662 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.662 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.662 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.662 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.662 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.662 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.662 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.662 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.663 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.663 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.663 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.663 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.663 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.663 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.663 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.663 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.663 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.663 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.663 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.663 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.663 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.664 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.664 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.664 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.664 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.664 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.664 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.664 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.664 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.664 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.664 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.664 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.664 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.664 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.665 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.665 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.665 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.665 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.665 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.665 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.665 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.665 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.665 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.665 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.665 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.665 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.665 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.666 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.666 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.666 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.666 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.666 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.666 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.666 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.666 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.666 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.666 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.666 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.666 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.666 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.667 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.667 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.667 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.667 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.667 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.667 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.667 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.667 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.667 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_asio 2024-03-11T20:53:53.667 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82522 2024-03-11T20:53:53.667 INFO:tasks.workunit.client.0.smithi129.stdout:test api_asio on pid 82522 2024-03-11T20:53:53.674 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_asio on pid 82522' 2024-03-11T20:53:53.674 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82522 2024-03-11T20:53:53.674 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.674 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.674 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-03-11T20:53:53.674 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_list 2024-03-11T20:53:53.675 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_list' 2024-03-11T20:53:53.675 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.675 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.675 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.675 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.675 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.675 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_list 2024-03-11T20:53:53.675 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.675 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.675 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.675 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.676 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.676 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.676 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.676 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.676 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.676 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.676 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.676 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.676 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.676 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.676 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.676 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.676 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.677 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.677 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.677 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.677 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.677 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.677 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.678 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.678 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.678 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.678 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.678 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.678 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.678 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.678 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.678 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.678 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.679 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.679 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.679 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.679 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.679 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.679 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.679 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.679 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.679 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.679 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.679 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.679 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.679 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.679 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.680 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.680 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.680 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.680 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.680 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.680 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.680 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.681 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.681 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.681 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.681 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.681 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.681 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.681 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.681 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.681 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.681 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.682 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.683 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.684 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.684 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.684 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.684 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.684 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_list 2024-03-11T20:53:53.684 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82540 2024-03-11T20:53:53.684 INFO:tasks.workunit.client.0.smithi129.stdout:test api_list on pid 82540 2024-03-11T20:53:53.692 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_list on pid 82540' 2024-03-11T20:53:53.692 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82540 2024-03-11T20:53:53.692 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.692 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.692 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-03-11T20:53:53.692 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.692 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82481/exe 2024-03-11T20:53:53.693 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_lock 2024-03-11T20:53:53.693 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_lock' 2024-03-11T20:53:53.693 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_lock 2024-03-11T20:53:53.693 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.693 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.693 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.693 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.693 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.693 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.693 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.693 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.694 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.694 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.694 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.694 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.694 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.694 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.694 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.694 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.694 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.694 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.695 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.695 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.695 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.695 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.695 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.695 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.695 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.695 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.695 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.695 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.695 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.695 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.695 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.695 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.696 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.696 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.696 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.696 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.696 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.696 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.696 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.696 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.696 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.696 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.696 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.696 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.696 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.696 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.696 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.697 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.698 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.698 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.698 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.698 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.698 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.698 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.698 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.698 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.698 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.698 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.698 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.698 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.698 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.699 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.699 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.699 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.699 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.699 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.699 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.699 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.699 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.699 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.699 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.699 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.699 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.699 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82488/exe 2024-03-11T20:53:53.699 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.700 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.700 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.700 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-03-11T20:53:53.700 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.700 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.700 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.700 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.700 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.700 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.700 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_aio: /' 2024-03-11T20:53:53.700 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.700 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.700 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.700 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.700 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.701 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.701 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.701 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.701 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_aio.log 2024-03-11T20:53:53.701 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.701 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_lock 2024-03-11T20:53:53.701 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82560 2024-03-11T20:53:53.701 INFO:tasks.workunit.client.0.smithi129.stdout:test api_lock on pid 82560 2024-03-11T20:53:53.705 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_lock on pid 82560' 2024-03-11T20:53:53.705 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82560 2024-03-11T20:53:53.705 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.705 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.705 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.705 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-03-11T20:53:53.705 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.705 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.705 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.705 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.705 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.706 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_lock_pp 2024-03-11T20:53:53.706 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.706 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_lock_pp' 2024-03-11T20:53:53.706 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.706 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.706 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.706 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.706 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.706 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.706 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.706 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_lock_pp 2024-03-11T20:53:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-03-11T20:53:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.707 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-03-11T20:53:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.708 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_aio_pp: /' 2024-03-11T20:53:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.709 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.710 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.710 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.710 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.710 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.711 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.711 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.711 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.711 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.711 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.711 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.711 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.711 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.711 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.711 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.711 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.711 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.711 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.711 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.712 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.712 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.712 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.712 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.712 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.712 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.712 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.712 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.712 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.712 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.712 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.712 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.713 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.713 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.713 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.713 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.713 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.713 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.713 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.713 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.713 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.713 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.713 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.713 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.714 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_lock_pp 2024-03-11T20:53:53.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.714 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.714 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.714 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.714 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.714 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.714 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.715 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.715 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.715 INFO:tasks.workunit.client.0.smithi129.stdout:test api_lock_pp on pid 82576 2024-03-11T20:53:53.718 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.718 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82576 2024-03-11T20:53:53.718 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_lock_pp on pid 82576' 2024-03-11T20:53:53.718 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82576 2024-03-11T20:53:53.718 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.719 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.719 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-03-11T20:53:53.719 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.719 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_misc 2024-03-11T20:53:53.719 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_misc' 2024-03-11T20:53:53.719 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82497/exe 2024-03-11T20:53:53.719 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.719 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.719 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.719 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.719 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.719 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.720 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.720 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.720 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.720 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.720 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.720 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.720 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.720 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.720 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.720 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.720 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.720 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.721 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.721 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.721 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.721 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.721 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.721 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.721 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.721 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.721 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.721 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.722 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.722 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.722 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.722 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.722 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.722 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.722 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.722 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.722 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_misc 2024-03-11T20:53:53.722 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.722 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.723 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.723 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.723 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.723 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.723 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.723 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.723 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.723 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.723 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.723 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.724 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.724 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.724 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.725 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.725 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.725 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.725 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.725 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.725 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.725 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.725 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.725 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.725 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.725 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.726 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.726 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.726 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.726 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.726 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.726 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.726 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.726 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.726 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.726 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.726 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.726 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.727 INFO:tasks.workunit.client.0.smithi129.stdout:test api_misc on pid 82599 2024-03-11T20:53:53.730 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.730 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.731 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.731 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.731 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_io: /' 2024-03-11T20:53:53.731 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.731 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.731 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.731 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.731 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.731 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.731 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.732 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.732 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.732 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_misc 2024-03-11T20:53:53.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.732 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.732 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.733 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82599 2024-03-11T20:53:53.733 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_misc on pid 82599' 2024-03-11T20:53:53.733 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82599 2024-03-11T20:53:53.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82508/exe 2024-03-11T20:53:53.733 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.733 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.733 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_misc_pp 2024-03-11T20:53:53.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.733 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.733 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_misc_pp' 2024-03-11T20:53:53.733 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_io.log 2024-03-11T20:53:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-03-11T20:53:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_misc_pp 2024-03-11T20:53:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.734 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.735 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.735 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.735 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.735 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.735 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.735 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.735 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.735 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.735 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.735 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.735 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-03-11T20:53:53.735 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.735 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.736 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.736 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.736 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.736 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.736 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.736 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.736 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.736 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.736 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.736 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.736 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.736 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.736 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.736 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.737 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.737 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.737 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.737 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.737 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.737 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.737 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.737 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.737 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.737 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.737 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.737 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.737 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_io_pp: /' 2024-03-11T20:53:53.738 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.738 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.738 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.738 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.738 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.738 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.738 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.738 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.738 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.738 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.738 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.738 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.739 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.739 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.739 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.739 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.739 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.739 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.739 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.739 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.739 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.739 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.739 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.739 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.739 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.739 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.740 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.740 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-03-11T20:53:53.740 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.740 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_misc_pp 2024-03-11T20:53:53.740 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82522/exe 2024-03-11T20:53:53.740 INFO:tasks.workunit.client.0.smithi129.stdout:test api_misc_pp on pid 82617 2024-03-11T20:53:53.745 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.745 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82617 2024-03-11T20:53:53.745 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_misc_pp on pid 82617' 2024-03-11T20:53:53.745 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82617 2024-03-11T20:53:53.745 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.745 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.745 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.745 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.745 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.745 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.746 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.746 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.746 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.746 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.746 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.746 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.746 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.746 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.746 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.746 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.746 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.747 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.747 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.747 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.747 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.747 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.747 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.747 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.747 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.747 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.747 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-03-11T20:53:53.747 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.747 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.748 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.748 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.748 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.748 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.748 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.748 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.748 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.748 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.748 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.748 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.748 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.748 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.749 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.749 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.749 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.749 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.749 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_tier_pp 2024-03-11T20:53:53.749 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.749 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.749 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.749 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.749 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.749 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.749 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.749 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.749 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_tier_pp' 2024-03-11T20:53:53.750 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.750 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.750 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.750 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-03-11T20:53:53.750 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_asio: /' 2024-03-11T20:53:53.750 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.750 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.750 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.750 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.750 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.750 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.750 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.750 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.751 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.751 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.751 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.751 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.751 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.751 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.751 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.751 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.751 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.751 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.751 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.751 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.751 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.751 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_asio.log 2024-03-11T20:53:53.752 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-03-11T20:53:53.752 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.752 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.752 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82540/exe 2024-03-11T20:53:53.752 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.752 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.752 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.752 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.752 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.752 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.752 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.752 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.752 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_tier_pp 2024-03-11T20:53:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.753 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.754 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.754 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.754 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.754 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.754 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.754 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.754 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.754 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.754 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.754 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.754 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.754 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.754 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.755 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.755 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.755 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.755 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.755 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.755 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.755 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.755 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.755 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.755 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.755 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.755 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.755 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.755 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.756 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.756 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.756 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.756 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.756 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.756 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.756 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.756 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.756 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-03-11T20:53:53.756 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.756 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.756 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.756 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.757 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.757 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_list.log 2024-03-11T20:53:53.757 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.757 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.757 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.757 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.757 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.757 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.757 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.757 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.758 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.758 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_tier_pp 2024-03-11T20:53:53.758 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82642 2024-03-11T20:53:53.758 INFO:tasks.workunit.client.0.smithi129.stdout:test api_tier_pp on pid 82642 2024-03-11T20:53:53.761 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_tier_pp on pid 82642' 2024-03-11T20:53:53.761 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82642 2024-03-11T20:53:53.761 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.761 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.761 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_list: /' 2024-03-11T20:53:53.761 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.761 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.761 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.761 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.761 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.761 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_pool 2024-03-11T20:53:53.762 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_pool' 2024-03-11T20:53:53.762 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_pool 2024-03-11T20:53:53.762 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-03-11T20:53:53.762 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.762 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.762 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.762 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.762 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.762 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.763 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.763 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.763 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.763 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.763 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.763 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.763 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.763 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.763 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.763 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.763 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.763 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.763 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.764 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.764 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.764 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.764 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.764 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.764 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.764 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.764 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.765 INFO:tasks.workunit.client.0.smithi129.stdout:test api_pool on pid 82651 2024-03-11T20:53:53.770 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_pool 2024-03-11T20:53:53.770 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82651 2024-03-11T20:53:53.770 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_pool on pid 82651' 2024-03-11T20:53:53.770 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82651 2024-03-11T20:53:53.771 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.771 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.771 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.771 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-03-11T20:53:53.771 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_snapshots 2024-03-11T20:53:53.771 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_snapshots' 2024-03-11T20:53:53.771 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.771 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.772 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.772 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.772 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.772 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.772 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.772 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.772 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.772 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.772 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.772 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.772 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.773 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.773 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.773 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.773 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.773 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.773 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.773 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.773 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.773 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.773 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.773 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.773 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.774 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.774 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.774 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.774 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.774 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.774 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.774 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.774 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.774 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.774 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.774 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.774 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.775 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.775 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.775 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.775 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.775 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.775 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.775 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.775 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.775 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.775 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.775 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.775 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.776 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.776 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.776 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.776 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.776 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.776 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.776 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.776 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.776 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.776 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.776 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.776 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.776 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.777 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.777 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.777 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.777 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.777 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.777 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.777 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.777 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.777 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.777 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.777 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.777 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.777 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.778 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.778 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.778 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.778 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.778 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.778 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.778 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.778 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.778 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.778 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.778 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.778 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.778 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.778 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.779 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.779 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.779 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.779 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.779 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.779 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.779 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.779 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.779 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.779 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.779 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.779 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.779 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.780 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.780 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.780 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.780 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.780 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.780 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.780 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.780 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.780 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.780 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.780 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.780 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.780 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.781 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.781 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.781 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.781 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.781 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.781 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.781 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.781 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.781 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.781 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.781 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.781 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.781 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.781 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.782 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.782 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.782 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.782 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.782 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.782 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.782 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.782 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.782 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.782 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.782 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.782 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82560/exe 2024-03-11T20:53:53.783 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.783 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.783 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82576/exe 2024-03-11T20:53:53.783 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.783 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.783 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.783 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.783 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.783 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.783 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.783 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.783 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.783 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.784 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.784 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.784 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.784 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.784 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.784 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.784 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.784 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.784 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.784 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.784 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.784 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.784 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_lock.log 2024-03-11T20:53:53.785 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_lock_pp: /' 2024-03-11T20:53:53.785 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-03-11T20:53:53.785 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-03-11T20:53:53.785 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.785 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.785 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.785 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.785 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.785 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.785 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.785 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.785 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.786 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.786 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.786 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.786 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.786 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.786 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.786 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.786 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.786 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.786 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.786 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.786 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.786 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.786 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.787 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.787 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.787 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.787 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-03-11T20:53:53.787 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.787 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_snapshots 2024-03-11T20:53:53.787 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.787 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.787 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.787 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.787 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.787 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.787 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_lock: /' 2024-03-11T20:53:53.787 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.787 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.788 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.788 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.788 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.788 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.788 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.788 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.788 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.788 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.788 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.788 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.788 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.788 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.788 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.788 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.788 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.788 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.789 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.789 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.789 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.789 INFO:tasks.workunit.client.0.smithi129.stdout:test api_snapshots on pid 82705 2024-03-11T20:53:53.793 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_snapshots 2024-03-11T20:53:53.793 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82705 2024-03-11T20:53:53.793 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_snapshots on pid 82705' 2024-03-11T20:53:53.793 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82705 2024-03-11T20:53:53.793 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.793 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.793 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-03-11T20:53:53.793 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.793 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.794 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.794 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.794 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.794 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.794 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.794 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.794 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.794 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.794 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.794 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.794 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.794 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.794 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.795 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.795 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.795 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.795 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.795 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.795 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.795 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.795 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.795 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.795 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.795 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.795 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.795 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.795 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.795 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.796 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.796 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.796 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.796 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.796 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.796 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.796 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.796 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.796 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.796 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.796 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.796 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_snapshots_pp 2024-03-11T20:53:53.796 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.796 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.797 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.797 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.797 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.797 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.797 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.797 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.797 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.797 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.797 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.797 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.797 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.797 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.797 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.798 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.798 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.798 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.798 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.798 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.798 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.798 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_snapshots_pp' 2024-03-11T20:53:53.798 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.798 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.798 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82617/exe 2024-03-11T20:53:53.798 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.798 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.798 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.799 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.799 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.799 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.799 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.799 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.799 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.799 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.799 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.799 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.799 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.799 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.799 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.799 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.799 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.799 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.800 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.800 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.800 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.800 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.800 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.800 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.800 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.800 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.800 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.800 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.800 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.800 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.801 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.801 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.801 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.801 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.801 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.801 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-03-11T20:53:53.801 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-03-11T20:53:53.801 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_misc_pp: /' 2024-03-11T20:53:53.801 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.801 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.802 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.802 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.802 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.802 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.802 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.802 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.802 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.802 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.802 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.802 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.802 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.802 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.802 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.803 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.803 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.803 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.803 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.803 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.803 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.803 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.803 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.803 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.803 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.803 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.803 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.804 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.804 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.804 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.804 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.804 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.804 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.804 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.804 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.804 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.804 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.804 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.805 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82599/exe 2024-03-11T20:53:53.805 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.805 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.805 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.805 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.805 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.805 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.805 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.805 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.805 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.805 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.805 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.806 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.806 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.806 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.806 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.806 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.806 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.806 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.806 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.807 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.807 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.807 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.807 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.807 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.807 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.807 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.807 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.807 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.807 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.807 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.807 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.807 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.807 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.808 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.808 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.808 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.808 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.808 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.808 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.808 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.808 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.808 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.808 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.809 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.809 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.809 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.809 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.809 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.809 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.809 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.809 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.809 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.809 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.809 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.809 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.809 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-03-11T20:53:53.809 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_snapshots_pp 2024-03-11T20:53:53.810 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.810 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_misc.log 2024-03-11T20:53:53.810 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.810 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_misc: /' 2024-03-11T20:53:53.810 INFO:tasks.workunit.client.0.smithi129.stdout:test api_snapshots_pp on pid 82794 2024-03-11T20:53:53.813 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_snapshots_pp 2024-03-11T20:53:53.813 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82794 2024-03-11T20:53:53.814 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_snapshots_pp on pid 82794' 2024-03-11T20:53:53.814 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82794 2024-03-11T20:53:53.814 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.814 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.814 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_stat 2024-03-11T20:53:53.814 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_stat' 2024-03-11T20:53:53.814 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-03-11T20:53:53.814 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.814 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.815 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.815 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.815 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.815 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.815 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.815 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.815 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.815 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.815 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.815 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.816 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.816 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.816 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.816 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.816 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.816 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.816 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.816 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.816 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.816 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.816 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.816 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.816 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.817 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.817 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.817 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.817 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.817 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.817 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.817 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.817 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.817 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.817 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.817 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.817 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.817 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.818 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.818 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.818 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.818 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.818 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.818 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.818 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.818 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.818 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.818 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.818 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.818 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.819 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.819 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.819 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.819 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.819 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.819 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.819 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.819 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.819 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.819 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.819 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.819 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.819 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.820 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.820 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.820 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.820 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.820 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.820 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.820 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.820 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.820 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.820 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.820 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.820 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.820 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.821 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.821 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.821 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.821 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.821 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.821 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.821 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.821 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82651/exe 2024-03-11T20:53:53.821 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_stat 2024-03-11T20:53:53.821 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82642/exe 2024-03-11T20:53:53.821 INFO:tasks.workunit.client.0.smithi129.stdout:test api_stat on pid 82816 2024-03-11T20:53:53.828 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_stat 2024-03-11T20:53:53.828 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82816 2024-03-11T20:53:53.828 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_stat on pid 82816' 2024-03-11T20:53:53.828 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82816 2024-03-11T20:53:53.828 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.828 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.828 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.828 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-03-11T20:53:53.828 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.829 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.829 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.829 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.829 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.829 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.829 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.829 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.829 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.829 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.829 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.829 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.829 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.829 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.829 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.830 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.830 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.830 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.830 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.830 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.830 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.830 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.830 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.830 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.830 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.830 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.830 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-03-11T20:53:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.831 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-03-11T20:53:53.832 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.832 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.832 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.832 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.832 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.832 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_pool: /' 2024-03-11T20:53:53.832 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.832 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.832 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.832 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.832 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.832 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.832 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.832 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.833 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.833 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.833 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.833 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.833 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.833 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.833 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_pool.log 2024-03-11T20:53:53.833 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.833 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.833 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.833 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.833 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.833 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.834 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.834 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.834 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.834 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.834 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.834 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.834 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.834 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.834 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.834 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.834 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-03-11T20:53:53.834 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.834 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.834 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.835 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.835 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.835 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.835 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.835 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.835 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.835 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.835 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.835 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.835 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.835 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.835 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.835 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.836 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.836 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.836 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.836 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_tier_pp: /' 2024-03-11T20:53:53.836 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_stat_pp 2024-03-11T20:53:53.836 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_stat_pp' 2024-03-11T20:53:53.836 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.836 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.836 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.836 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.836 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.836 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.836 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.836 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.836 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.837 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.837 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.837 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.837 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.837 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.837 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.837 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.837 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.837 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.837 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.837 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.837 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.837 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.838 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.839 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.839 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.839 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.839 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.839 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.839 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.839 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.839 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.839 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.839 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.839 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.839 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.839 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.839 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.839 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.840 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.840 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.840 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.840 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.840 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.840 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.840 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.840 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.840 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.840 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.840 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.841 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.841 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.841 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.841 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.841 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.841 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.841 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.841 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.841 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.841 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.841 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.841 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.841 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.842 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.842 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.842 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.842 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.842 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.842 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.842 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.842 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.842 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.842 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.842 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.842 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.843 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.843 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.843 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.843 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.843 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.843 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.843 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_stat_pp 2024-03-11T20:53:53.843 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.843 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.843 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.843 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.844 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.844 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.844 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.844 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.844 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.844 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.844 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.844 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.844 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.844 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.844 INFO:tasks.workunit.client.0.smithi129.stdout:test api_stat_pp on pid 82899 2024-03-11T20:53:53.851 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_stat_pp 2024-03-11T20:53:53.852 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82899 2024-03-11T20:53:53.852 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_stat_pp on pid 82899' 2024-03-11T20:53:53.852 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82899 2024-03-11T20:53:53.852 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.852 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.852 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.852 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.852 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.852 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.852 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.853 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.853 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.853 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.853 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.853 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.853 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.853 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.853 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.853 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.853 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.854 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.854 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.854 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.854 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.854 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.854 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.854 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.854 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.854 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.855 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.855 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.855 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.855 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.855 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.855 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.855 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.855 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.855 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.855 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.855 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.855 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.855 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.856 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.856 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_watch_notify 2024-03-11T20:53:53.856 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.856 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.856 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.856 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.856 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_watch_notify' 2024-03-11T20:53:53.856 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-03-11T20:53:53.856 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82794/exe 2024-03-11T20:53:53.856 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_watch_notify 2024-03-11T20:53:53.856 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82816/exe 2024-03-11T20:53:53.856 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.856 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.857 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.857 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.857 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.857 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.857 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.857 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.857 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.857 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.857 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.857 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.857 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.857 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.858 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.858 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.858 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.858 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.858 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.858 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.858 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-03-11T20:53:53.858 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.858 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.858 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.858 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.858 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.858 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.859 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-03-11T20:53:53.859 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.859 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.859 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.859 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.859 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.859 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.859 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.859 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.859 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.859 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.859 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.859 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.859 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-03-11T20:53:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.860 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.861 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.862 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.862 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.862 INFO:tasks.workunit.client.0.smithi129.stdout:test api_watch_notify on pid 82925 2024-03-11T20:53:53.864 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_watch_notify 2024-03-11T20:53:53.864 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82925 2024-03-11T20:53:53.864 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_watch_notify on pid 82925' 2024-03-11T20:53:53.864 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82925 2024-03-11T20:53:53.865 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.865 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.865 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-03-11T20:53:53.865 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82705/exe 2024-03-11T20:53:53.865 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-03-11T20:53:53.865 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.865 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.865 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.865 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.865 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.865 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.865 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_stat.log 2024-03-11T20:53:53.865 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.865 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.866 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.866 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.866 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.866 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.866 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.866 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.866 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.866 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.866 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_watch_notify_pp 2024-03-11T20:53:53.866 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_watch_notify_pp' 2024-03-11T20:53:53.866 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_stat: /' 2024-03-11T20:53:53.866 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.866 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.867 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.867 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.867 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.867 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.867 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.867 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.867 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.867 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.867 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_watch_notify_pp 2024-03-11T20:53:53.867 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_snapshots: /' 2024-03-11T20:53:53.867 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-03-11T20:53:53.867 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-03-11T20:53:53.867 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.867 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.868 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.868 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.868 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.868 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.868 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.868 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.868 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.868 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.868 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.868 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.868 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.868 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.868 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.869 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.869 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.869 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.869 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.869 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.869 INFO:tasks.workunit.client.0.smithi129.stdout:test api_watch_notify_pp on pid 82954 2024-03-11T20:53:53.870 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_watch_notify_pp 2024-03-11T20:53:53.871 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82954 2024-03-11T20:53:53.871 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_watch_notify_pp on pid 82954' 2024-03-11T20:53:53.871 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82954 2024-03-11T20:53:53.871 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.871 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.871 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_cmd 2024-03-11T20:53:53.871 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_cmd' 2024-03-11T20:53:53.871 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.871 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.871 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.871 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-03-11T20:53:53.871 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.871 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.872 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.872 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.872 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.872 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.872 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.872 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.872 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.872 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.872 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.872 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.872 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.872 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.872 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_cmd 2024-03-11T20:53:53.873 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.873 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.873 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.873 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.873 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.873 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.873 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.873 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.873 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.873 INFO:tasks.workunit.client.0.smithi129.stdout:test api_cmd on pid 82970 2024-03-11T20:53:53.877 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_cmd 2024-03-11T20:53:53.877 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82970 2024-03-11T20:53:53.877 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_cmd on pid 82970' 2024-03-11T20:53:53.877 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=82970 2024-03-11T20:53:53.877 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.877 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.877 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-03-11T20:53:53.877 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.877 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.877 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.877 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.877 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.878 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.878 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.878 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.878 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.878 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.878 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.878 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.878 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.878 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.878 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.878 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.878 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.879 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.879 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.879 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.879 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.879 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.879 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.879 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.879 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.879 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.879 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.880 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.880 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.880 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.880 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.880 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.880 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.880 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.880 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.880 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.880 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.880 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.880 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.880 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.880 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.881 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.881 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.881 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.881 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_cmd_pp 2024-03-11T20:53:53.881 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.881 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.881 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.881 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.881 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.881 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_cmd_pp' 2024-03-11T20:53:53.881 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.881 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.882 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.882 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.882 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.882 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.882 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.882 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.882 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.882 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.882 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.882 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.882 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.883 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.883 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.883 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.883 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.883 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.883 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.883 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.883 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.883 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.883 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.883 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.884 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.884 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.884 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.884 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.884 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.884 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.884 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.884 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.884 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.884 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.884 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_cmd_pp 2024-03-11T20:53:53.884 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.884 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.884 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.884 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.885 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.885 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.885 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.885 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.885 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.885 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.885 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.885 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.885 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.885 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.885 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.885 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.885 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.885 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.886 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.886 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.886 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.886 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.886 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.886 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.886 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.886 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.886 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.886 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.886 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.886 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.886 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.886 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.887 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.887 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.887 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.887 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.887 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.887 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.887 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.887 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.887 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.887 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.887 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.888 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.888 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.888 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.888 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.888 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82899/exe 2024-03-11T20:53:53.888 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.888 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.888 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.888 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.888 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.888 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.888 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.889 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.889 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.889 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.889 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.889 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.889 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.889 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.889 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.889 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.889 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.890 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.890 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.890 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.890 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-03-11T20:53:53.890 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.890 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.890 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.890 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.890 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.890 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.890 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.890 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.890 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.890 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.891 INFO:tasks.workunit.client.0.smithi129.stdout:test api_cmd_pp on pid 83020 2024-03-11T20:53:53.894 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.894 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_cmd_pp 2024-03-11T20:53:53.894 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.894 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.894 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.894 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.895 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.895 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.895 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.895 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.895 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83020 2024-03-11T20:53:53.895 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_cmd_pp on pid 83020' 2024-03-11T20:53:53.895 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83020 2024-03-11T20:53:53.895 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.895 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.896 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.896 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.896 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.896 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.896 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.896 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.896 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.896 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.896 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.896 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.897 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.897 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.897 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_service 2024-03-11T20:53:53.897 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.897 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.897 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.897 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.897 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.897 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_service' 2024-03-11T20:53:53.897 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-03-11T20:53:53.897 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_stat_pp: /' 2024-03-11T20:53:53.897 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-03-11T20:53:53.897 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.898 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_service 2024-03-11T20:53:53.898 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.898 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.898 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.898 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.898 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.898 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.898 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.898 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.898 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.898 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.898 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.899 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.899 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.899 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.899 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.899 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.899 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.899 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.899 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.899 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.899 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.899 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.899 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.900 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.900 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.900 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.900 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.900 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.900 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.900 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.900 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.900 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.900 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.900 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.900 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.900 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.900 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.901 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.902 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.903 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.903 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.903 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.903 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.903 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.903 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.903 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.903 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.903 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.903 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.903 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.903 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.903 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.903 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.904 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.904 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.904 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.904 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.904 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.904 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.904 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.904 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.904 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.904 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.905 INFO:tasks.workunit.client.0.smithi129.stdout:test api_service on pid 83049 2024-03-11T20:53:53.906 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_service 2024-03-11T20:53:53.907 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83049 2024-03-11T20:53:53.907 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_service on pid 83049' 2024-03-11T20:53:53.907 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83049 2024-03-11T20:53:53.907 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.907 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.907 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.907 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.907 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.907 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-03-11T20:53:53.907 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.907 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.907 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_service_pp 2024-03-11T20:53:53.907 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.907 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.908 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.908 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.908 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.908 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_service_pp' 2024-03-11T20:53:53.908 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.908 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.908 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.908 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.908 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.908 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.908 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.908 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.908 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_service_pp 2024-03-11T20:53:53.908 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.909 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.909 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.909 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.909 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.909 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.909 INFO:tasks.workunit.client.0.smithi129.stdout:test api_service_pp on pid 83062 2024-03-11T20:53:53.913 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_service_pp 2024-03-11T20:53:53.913 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.914 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.914 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.914 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83062 2024-03-11T20:53:53.914 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.914 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.914 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_service_pp on pid 83062' 2024-03-11T20:53:53.914 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.914 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83062 2024-03-11T20:53:53.914 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.914 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.914 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.914 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.914 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.915 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.916 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.916 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.916 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.916 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.916 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.916 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.916 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.916 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.916 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.916 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.916 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.916 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.916 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.916 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.917 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.917 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.917 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.917 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.917 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.917 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.917 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.917 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.917 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.917 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.917 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.917 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.917 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.917 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.918 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.918 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.918 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.918 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.918 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-03-11T20:53:53.918 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.918 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.918 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.918 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.918 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.918 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.918 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.918 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.919 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_c_write_operations 2024-03-11T20:53:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_c_write_operations' 2024-03-11T20:53:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.920 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.921 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.921 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.921 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.921 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.921 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.921 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.921 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.921 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.921 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.921 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82970/exe 2024-03-11T20:53:53.921 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.922 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.922 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.922 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.922 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.922 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.922 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.922 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.922 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.922 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.922 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.922 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.922 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.922 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_c_write_operations 2024-03-11T20:53:53.923 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.923 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.923 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.923 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.923 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.923 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.923 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.923 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.923 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.923 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.924 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.924 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.924 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.924 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.924 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.924 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.924 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.924 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.924 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.924 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.924 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.924 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.924 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.925 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.925 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.925 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.925 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.925 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.925 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.925 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82954/exe 2024-03-11T20:53:53.925 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.925 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.925 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.926 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.926 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.926 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.926 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.926 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.926 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.926 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.926 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_c_write_operations 2024-03-11T20:53:53.927 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.927 INFO:tasks.workunit.client.0.smithi129.stdout:test api_c_write_operations on pid 83100 2024-03-11T20:53:53.930 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83100 2024-03-11T20:53:53.931 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_c_write_operations on pid 83100' 2024-03-11T20:53:53.931 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83100 2024-03-11T20:53:53.931 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.931 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.931 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_cmd.log 2024-03-11T20:53:53.931 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/82925/exe 2024-03-11T20:53:53.931 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_c_read_operations 2024-03-11T20:53:53.931 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-03-11T20:53:53.931 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_c_read_operations' 2024-03-11T20:53:53.932 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-03-11T20:53:53.932 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.932 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.932 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.932 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.932 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.932 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.932 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.932 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.932 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_cmd: /' 2024-03-11T20:53:53.933 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.933 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.933 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.933 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.933 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.933 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.933 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.933 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.933 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.934 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.934 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.934 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.934 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.934 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.934 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.934 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.934 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.934 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.934 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.935 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-03-11T20:53:53.935 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.935 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.935 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.935 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.935 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.935 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.935 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.935 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.935 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.935 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.936 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.936 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.936 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.936 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_c_read_operations 2024-03-11T20:53:53.936 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.936 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.936 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.936 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.936 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.936 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.936 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.936 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.937 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.937 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-03-11T20:53:53.937 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.937 INFO:tasks.workunit.client.0.smithi129.stdout:test api_c_read_operations on pid 83115 2024-03-11T20:53:53.943 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_c_read_operations 2024-03-11T20:53:53.943 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.943 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.943 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.944 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.944 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83115 2024-03-11T20:53:53.944 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_c_read_operations on pid 83115' 2024-03-11T20:53:53.944 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83115 2024-03-11T20:53:53.944 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.944 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.944 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.944 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-03-11T20:53:53.944 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.944 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.944 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.944 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.944 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.945 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s api_cls_remote_reads 2024-03-11T20:53:53.945 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.945 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.945 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.945 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.945 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.945 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-03-11T20:53:53.945 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-03-11T20:53:53.945 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.945 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' api_cls_remote_reads' 2024-03-11T20:53:53.945 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.945 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.945 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.946 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.946 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.946 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.946 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.946 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.946 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.946 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.946 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.946 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.946 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.946 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.946 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.946 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.947 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.947 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.947 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.947 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.947 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.947 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.947 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-03-11T20:53:53.947 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.947 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.947 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.947 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.947 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.947 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_watch_notify: /' 2024-03-11T20:53:53.947 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.948 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.948 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.948 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.948 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.948 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.948 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.948 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.948 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.948 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.948 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.948 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.948 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.949 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.949 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.949 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.949 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.949 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.949 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.949 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.949 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.949 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.949 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.949 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.949 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.950 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.950 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.950 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.950 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.950 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.950 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.950 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.950 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.950 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.950 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.950 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.950 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.951 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.951 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.951 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.951 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.951 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.951 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.951 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.951 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.951 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.951 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.951 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.951 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.951 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.951 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.952 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.952 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.952 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.952 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.952 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.952 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.952 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.952 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.952 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.952 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.952 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.952 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.952 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.952 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.953 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.953 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.953 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.953 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.953 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.953 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.953 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.953 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.953 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.953 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.953 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.953 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.953 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.953 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.954 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.954 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.954 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.954 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.954 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.954 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.954 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.954 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83020/exe 2024-03-11T20:53:53.954 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.954 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.954 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.954 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.954 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.954 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.955 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.955 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.955 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.955 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.955 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.955 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.955 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.955 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.955 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.955 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.955 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.955 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.955 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.955 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.956 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83049/exe 2024-03-11T20:53:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-03-11T20:53:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-03-11T20:53:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-03-11T20:53:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.957 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.958 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.959 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.959 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.959 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.959 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.959 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.959 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo api_cls_remote_reads 2024-03-11T20:53:53.959 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.959 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.959 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.959 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.959 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.959 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.959 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.959 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.960 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.960 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.960 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.960 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_service.log 2024-03-11T20:53:53.960 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.960 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.960 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.960 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.960 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.960 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.960 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.960 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.961 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.961 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.961 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.961 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_service: /' 2024-03-11T20:53:53.961 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.961 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.961 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.961 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.961 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.961 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.961 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.962 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.962 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.962 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.962 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.962 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.962 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.962 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.962 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.962 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.962 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.962 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.962 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.962 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.963 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.963 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.963 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.963 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.963 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.963 INFO:tasks.workunit.client.0.smithi129.stdout:test api_cls_remote_reads on pid 83173 2024-03-11T20:53:53.965 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=api_cls_remote_reads 2024-03-11T20:53:53.965 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83173 2024-03-11T20:53:53.965 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test api_cls_remote_reads on pid 83173' 2024-03-11T20:53:53.965 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83173 2024-03-11T20:53:53.965 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.965 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.966 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-03-11T20:53:53.966 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.966 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.966 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.966 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.966 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.966 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.966 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.966 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.966 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.966 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.966 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.967 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.967 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.967 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.967 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.967 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.967 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.967 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.967 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.967 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.967 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.967 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.967 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.967 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.967 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.967 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.968 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.968 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.968 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.968 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.968 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.968 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.968 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.968 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.968 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.968 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.968 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s list_parallel 2024-03-11T20:53:53.968 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.968 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.968 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.969 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.969 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.969 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.969 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.969 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.969 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.969 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' list_parallel' 2024-03-11T20:53:53.969 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.969 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.969 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.969 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.969 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.969 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.970 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.970 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.970 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.970 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.970 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.970 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.970 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.970 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.970 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.970 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.971 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.971 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.971 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.971 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.971 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.971 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-03-11T20:53:53.971 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo list_parallel 2024-03-11T20:53:53.971 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.971 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.971 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.971 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.972 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.972 INFO:tasks.workunit.client.0.smithi129.stdout:test list_parallel on pid 83188 2024-03-11T20:53:53.977 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=list_parallel 2024-03-11T20:53:53.977 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83188 2024-03-11T20:53:53.977 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test list_parallel on pid 83188' 2024-03-11T20:53:53.977 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83188 2024-03-11T20:53:53.977 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.977 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:53.977 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.977 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:53.977 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:53.977 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.978 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:53.978 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.978 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:53.978 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:53.978 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:53.978 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.978 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:53.978 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.978 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:53.978 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:53.978 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:53.978 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.979 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:53.979 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.979 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:53.979 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.979 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s open_pools_parallel 2024-03-11T20:53:53.979 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-03-11T20:53:53.979 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' open_pools_parallel' 2024-03-11T20:53:53.979 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.979 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.979 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.979 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.979 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.979 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.979 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo open_pools_parallel 2024-03-11T20:53:53.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.980 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.980 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.980 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.980 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.981 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.981 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.981 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.981 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.981 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.981 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.982 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.982 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.982 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.982 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:53.982 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:53.982 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:53.982 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.982 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:53.982 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.982 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:53.982 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:53.982 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.982 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83062/exe 2024-03-11T20:53:53.982 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.983 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.983 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.983 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.983 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.983 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.984 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.984 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.984 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.984 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.984 INFO:tasks.workunit.client.0.smithi129.stdout:test open_pools_parallel on pid 83208 2024-03-11T20:53:53.986 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.986 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.986 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.986 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=open_pools_parallel 2024-03-11T20:53:53.986 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.986 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.986 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.986 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.986 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.987 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.987 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83208 2024-03-11T20:53:53.987 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test open_pools_parallel on pid 83208' 2024-03-11T20:53:53.987 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83208 2024-03-11T20:53:53.987 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-11T20:53:53.987 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.987 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.987 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.987 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.987 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.987 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.987 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.987 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.987 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.987 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s delete_pools_parallel 2024-03-11T20:53:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' delete_pools_parallel' 2024-03-11T20:53:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.988 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-03-11T20:53:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83100/exe 2024-03-11T20:53:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo delete_pools_parallel 2024-03-11T20:53:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:53.989 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:53.990 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:53.990 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:53.990 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:53.990 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:53.990 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:53.990 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:53.990 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_service_pp: /' 2024-03-11T20:53:53.990 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.990 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:53.990 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.990 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:53.990 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:53.990 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=delete_pools_parallel 2024-03-11T20:53:53.990 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83234 2024-03-11T20:53:53.991 INFO:tasks.workunit.client.0.smithi129.stdout:test delete_pools_parallel on pid 83234 2024-03-11T20:53:53.994 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test delete_pools_parallel on pid 83234' 2024-03-11T20:53:53.994 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83234 2024-03-11T20:53:53.994 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-03-11T20:53:53.994 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T20:53:53.994 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:53.994 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:53.994 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:53.995 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:53.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:53.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:53.995 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:53.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:53.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s cls 2024-03-11T20:53:53.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:53.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:53.995 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:53.996 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' cls' 2024-03-11T20:53:53.996 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:53.996 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:53.996 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:53.996 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:53.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:53.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:53.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:53.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.996 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:53.996 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:53.996 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:53.996 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:53.997 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:53.997 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:53.997 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:53.997 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.997 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:53.997 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:53.997 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:53.997 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:53.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:53.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:53.997 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:53.997 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:53.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.997 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:53.998 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.998 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:53.998 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.998 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:53.998 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.998 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:53.998 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:53.998 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.998 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:53.998 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.998 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:53.998 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:53.998 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-03-11T20:53:53.998 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:53.999 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.999 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:53.999 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.999 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:53.999 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.999 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:53.999 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:53.999 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:53.999 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:53.999 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:53.999 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:53.999 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-03-11T20:53:54.000 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo cls 2024-03-11T20:53:54.000 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-03-11T20:53:54.000 INFO:tasks.workunit.client.0.smithi129.stdout:test cls on pid 83255 2024-03-11T20:53:54.008 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=cls 2024-03-11T20:53:54.008 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83255 2024-03-11T20:53:54.009 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test cls on pid 83255' 2024-03-11T20:53:54.009 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83255 2024-03-11T20:53:54.009 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T20:53:54.009 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:54.009 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:54.009 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:54.009 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:54.009 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.009 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.009 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.010 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.010 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.010 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.010 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s cmd 2024-03-11T20:53:54.010 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:54.010 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.010 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:54.010 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:54.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:54.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:54.011 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' cmd' 2024-03-11T20:53:54.011 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-03-11T20:53:54.011 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.011 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:54.011 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.011 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:54.012 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.012 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:54.012 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:54.012 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:54.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:54.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.012 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:54.012 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:54.013 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:54.013 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.013 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:54.013 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.013 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:54.013 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:54.013 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:54.013 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.013 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:54.013 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:54.013 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.014 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:54.014 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:54.014 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:54.014 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.014 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.014 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:54.014 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.014 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:54.014 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:54.014 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:54.015 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.015 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:54.015 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.015 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:54.015 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:54.015 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:54.015 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.015 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:54.015 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.015 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:54.015 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.015 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-03-11T20:53:54.015 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:54.015 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:54.016 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:54.016 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:54.016 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.016 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:54.016 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-03-11T20:53:54.016 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83173/exe 2024-03-11T20:53:54.016 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.016 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.016 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.016 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.016 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.016 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.016 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.016 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.017 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.017 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.017 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.017 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:54.017 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:54.017 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:54.017 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.017 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:54.017 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.017 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:54.017 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:54.017 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.017 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-03-11T20:53:54.017 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:54.017 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:54.018 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:54.018 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.018 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:54.018 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.018 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:54.018 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:54.018 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:54.018 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.018 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:54.018 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.018 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:54.018 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:54.018 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:54.018 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.019 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:54.019 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.019 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:54.019 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.019 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.019 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.019 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.019 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.019 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.019 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.019 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.019 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.019 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.019 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.020 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.020 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.020 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.020 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.020 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.020 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.020 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.020 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.020 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.020 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.020 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo cmd 2024-03-11T20:53:54.020 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.020 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:54.020 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-03-11T20:53:54.020 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:54.020 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:54.021 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:54.021 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.021 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:54.021 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.021 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:54.021 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:54.021 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.021 INFO:tasks.workunit.client.0.smithi129.stdout:test cmd on pid 83290 2024-03-11T20:53:54.023 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:54.023 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:54.023 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=cmd 2024-03-11T20:53:54.023 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:54.023 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.023 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:54.023 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.023 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:54.023 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:54.023 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.024 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83290 2024-03-11T20:53:54.024 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test cmd on pid 83290' 2024-03-11T20:53:54.024 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83290 2024-03-11T20:53:54.024 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T20:53:54.024 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:54.024 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-03-11T20:53:54.024 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s handler_error 2024-03-11T20:53:54.024 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' handler_error' 2024-03-11T20:53:54.024 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:54.024 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:54.024 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:54.024 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:54.024 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:54.024 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-03-11T20:53:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.026 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.026 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.026 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:54.026 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.026 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:54.026 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.026 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo handler_error 2024-03-11T20:53:54.026 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=handler_error 2024-03-11T20:53:54.026 INFO:tasks.workunit.client.0.smithi129.stdout:test handler_error on pid 83317 2024-03-11T20:53:54.033 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83317 2024-03-11T20:53:54.033 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test handler_error on pid 83317' 2024-03-11T20:53:54.033 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83317 2024-03-11T20:53:54.033 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T20:53:54.033 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:54.033 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:54.033 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:54.033 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:54.033 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.033 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:54.034 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.034 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:54.034 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:54.034 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.034 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:54.034 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.034 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:54.034 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.034 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:54.034 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.034 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:54.034 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:54.035 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:54.035 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:54.035 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:54.035 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.035 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:54.035 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.035 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:54.035 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:54.035 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.035 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:54.035 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.035 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:54.035 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.036 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.036 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.036 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:54.036 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.036 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:54.036 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.036 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:54.036 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s io 2024-03-11T20:53:54.036 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:54.036 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.037 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:54.037 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.037 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:54.037 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:54.037 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:54.037 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:54.037 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.037 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.037 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:54.037 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.037 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.037 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.037 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:54.038 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:54.038 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.038 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.038 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:54.038 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:54.038 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.038 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:54.038 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.038 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:54.038 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:54.039 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.039 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:54.039 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.039 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:54.039 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.039 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:54.039 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.039 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:54.039 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:54.039 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-03-11T20:53:54.039 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.040 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:54.040 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.040 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.040 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:54.040 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:54.040 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83115/exe 2024-03-11T20:53:54.040 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' io' 2024-03-11T20:53:54.040 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:54.040 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:54.040 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:54.040 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:54.040 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:54.040 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.041 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:54.041 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:54.041 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:54.041 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:54.041 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.041 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.041 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:54.041 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.041 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.041 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.042 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:54.042 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.042 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.042 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.042 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.042 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.042 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.042 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.042 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.042 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:54.042 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.042 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.042 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:54.043 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.043 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.043 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:54.043 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.043 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.043 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.043 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.043 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:54.043 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:54.044 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.044 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.044 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.044 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.044 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.045 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.045 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.045 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.045 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:54.045 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.045 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.045 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.045 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.046 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.046 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.046 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.046 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.046 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.046 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.046 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.046 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.046 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.046 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.046 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.046 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.046 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.047 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.047 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.047 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.047 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.047 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.047 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.047 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.047 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.047 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.047 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.047 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.047 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.047 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.047 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.048 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.048 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.048 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.048 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.048 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.048 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.048 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-03-11T20:53:54.048 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-03-11T20:53:54.048 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.048 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.048 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.048 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.048 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.048 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.049 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.049 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.049 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.049 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.049 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.049 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.049 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.049 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.049 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.049 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.049 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.049 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.049 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83208/exe 2024-03-11T20:53:54.049 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.050 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.050 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.050 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.050 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83234/exe 2024-03-11T20:53:54.050 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.050 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.050 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.050 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.050 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.050 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.050 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.050 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.050 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.051 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.051 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-03-11T20:53:54.051 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-03-11T20:53:54.051 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:54.051 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:54.051 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:54.051 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.051 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:54.052 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.052 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:54.052 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:54.052 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:54.052 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.052 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:54.052 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.052 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:54.052 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:54.052 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:54.053 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:54.053 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.053 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:54.053 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.053 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:54.053 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:54.053 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.053 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:54.053 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.053 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:54.053 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.054 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:54.054 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:54.054 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:54.054 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.054 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:54.054 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.054 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:54.054 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:54.054 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:54.055 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.055 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:54.055 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.055 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:54.055 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.055 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.055 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:54.055 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:54.055 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:54.055 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.055 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:54.055 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.055 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:54.055 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:54.056 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:54.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:54.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:54.056 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:54.056 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:54.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:54.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.056 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:54.056 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.056 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.056 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.056 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-03-11T20:53:54.057 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.057 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.057 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.057 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.057 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.057 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.057 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.057 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.057 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-03-11T20:53:54.057 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83188/exe 2024-03-11T20:53:54.057 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.057 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-03-11T20:53:54.057 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.058 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:54.058 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.058 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.058 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.058 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.058 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.058 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.058 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo io 2024-03-11T20:53:54.058 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.058 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.058 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.058 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-03-11T20:53:54.059 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-03-11T20:53:54.059 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:54.059 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:54.059 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:54.059 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.059 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:54.059 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.059 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:54.059 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:54.059 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.059 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:54.059 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:54.059 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:54.059 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.059 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:54.060 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.060 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:54.060 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:54.060 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.060 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:54.060 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.060 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:54.060 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.060 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:54.060 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.060 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:54.060 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:54.060 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:54.061 INFO:tasks.workunit.client.0.smithi129.stdout:test io on pid 83376 2024-03-11T20:53:54.063 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=io 2024-03-11T20:53:54.063 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83376 2024-03-11T20:53:54.063 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test io on pid 83376' 2024-03-11T20:53:54.063 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83376 2024-03-11T20:53:54.063 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T20:53:54.063 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:54.064 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.064 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.064 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.064 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.064 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.064 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.064 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.064 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.064 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.064 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.064 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.064 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.064 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.064 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.065 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.065 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.065 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.065 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.065 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.065 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.065 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.065 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.065 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.065 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.065 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.065 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.065 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.065 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83255/exe 2024-03-11T20:53:54.066 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:54.066 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:54.066 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:54.066 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.066 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:54.066 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.066 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:54.066 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:54.066 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.066 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s ec_io 2024-03-11T20:53:54.066 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' ec_io' 2024-03-11T20:53:54.066 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.066 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.067 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.067 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.067 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.067 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.067 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.067 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.067 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.067 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.067 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.067 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-03-11T20:53:54.067 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-03-11T20:53:54.067 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ cls: /' 2024-03-11T20:53:54.067 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:54.068 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_rados_list_parallel.log 2024-03-11T20:53:54.068 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:54.068 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:54.068 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:54.068 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.068 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo ec_io 2024-03-11T20:53:54.068 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:54.068 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:54.068 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_cls 2024-03-11T20:53:54.068 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ list_parallel: /' 2024-03-11T20:53:54.068 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_cls.log 2024-03-11T20:53:54.069 INFO:tasks.workunit.client.0.smithi129.stdout:test ec_io on pid 83410 2024-03-11T20:53:54.070 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=ec_io 2024-03-11T20:53:54.070 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83410 2024-03-11T20:53:54.070 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test ec_io on pid 83410' 2024-03-11T20:53:54.070 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83410 2024-03-11T20:53:54.070 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T20:53:54.071 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:54.071 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s list 2024-03-11T20:53:54.071 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' list' 2024-03-11T20:53:54.071 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-03-11T20:53:54.071 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:54.071 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:54.071 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:54.071 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:54.071 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:54.071 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.071 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:54.071 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.071 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:54.071 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:54.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:54.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:54.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:54.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:54.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:54.072 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.072 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:54.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.072 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:54.073 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.073 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:54.073 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:54.073 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:54.073 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:54.073 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.073 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:54.073 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.073 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:54.073 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:54.073 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.073 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:54.073 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.073 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:54.073 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:54.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:54.074 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:54.074 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:54.074 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:54.074 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:54.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:54.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.074 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:54.074 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:54.074 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.075 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:54.075 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:54.075 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:54.075 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.075 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:54.075 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.075 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:54.075 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:54.075 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.075 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:54.075 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.075 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:54.075 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.075 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.076 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.076 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:54.076 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.076 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:54.076 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.076 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo list 2024-03-11T20:53:54.076 INFO:tasks.workunit.client.0.smithi129.stdout:test list on pid 83441 2024-03-11T20:53:54.080 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=list 2024-03-11T20:53:54.080 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83441 2024-03-11T20:53:54.080 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test list on pid 83441' 2024-03-11T20:53:54.080 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83441 2024-03-11T20:53:54.080 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T20:53:54.080 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:54.080 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s ec_list 2024-03-11T20:53:54.080 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' ec_list' 2024-03-11T20:53:54.080 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:54.080 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-03-11T20:53:54.080 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:54.080 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:54.081 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:54.081 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:54.081 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:54.081 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:54.081 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.081 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.081 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:54.081 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:54.081 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.081 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.081 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:54.081 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:54.081 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:54.082 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:54.082 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:54.082 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.082 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.082 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:54.082 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:54.082 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.082 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.082 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:54.083 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:54.083 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:54.083 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:54.083 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.083 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:54.083 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.083 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:54.083 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.083 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.083 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:54.083 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.083 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:54.083 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:54.083 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:54.084 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:54.084 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:54.084 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:54.084 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:54.084 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.084 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:54.084 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo ec_list 2024-03-11T20:53:54.084 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:54.084 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:54.084 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:54.085 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.085 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:54.085 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.085 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:54.085 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:54.085 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.085 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:54.085 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.085 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:54.085 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.085 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.085 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.085 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:54.086 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.086 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:54.086 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.086 INFO:tasks.workunit.client.0.smithi129.stdout:test ec_list on pid 83454 2024-03-11T20:53:54.091 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=ec_list 2024-03-11T20:53:54.091 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83454 2024-03-11T20:53:54.091 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test ec_list on pid 83454' 2024-03-11T20:53:54.091 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83454 2024-03-11T20:53:54.091 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T20:53:54.092 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:54.092 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s misc 2024-03-11T20:53:54.092 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' misc' 2024-03-11T20:53:54.092 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-03-11T20:53:54.092 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:54.092 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.092 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.092 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.092 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.092 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.092 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.093 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:54.093 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:54.093 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:54.093 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.093 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:54.093 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.093 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:54.093 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:54.093 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.094 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.094 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.094 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.094 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.094 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.094 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.094 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.094 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.094 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.094 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.094 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.094 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.095 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.095 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.095 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.095 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.095 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.095 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.095 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.095 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.095 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.095 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.095 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.095 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.095 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.095 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.096 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:54.096 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:54.096 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:54.096 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:54.096 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.096 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:54.096 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.096 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.096 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.096 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.096 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.096 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.096 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.097 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.097 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.097 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.097 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.097 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.097 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.097 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.097 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.097 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.097 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.097 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.097 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.097 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.098 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.098 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.098 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:54.098 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:54.098 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:54.098 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.098 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:54.098 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.098 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:54.098 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:54.098 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:54.098 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.098 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:54.098 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.099 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:54.099 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:54.099 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:54.099 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.099 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:54.099 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.099 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:54.099 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.099 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83317/exe 2024-03-11T20:53:54.099 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.099 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.099 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.099 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.099 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.100 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.100 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.100 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.100 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:54.100 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.100 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.100 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.100 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:54.100 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:54.100 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.100 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:54.100 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.100 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:54.100 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:54.101 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.101 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:54.101 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.101 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:54.101 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.101 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.101 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.101 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ handler_error: /' 2024-03-11T20:53:54.101 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:54.101 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.101 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:54.101 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.101 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_handler_error.log 2024-03-11T20:53:54.101 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_handler_error 2024-03-11T20:53:54.102 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:54.102 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:54.102 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:54.102 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.102 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:54.102 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.102 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:54.102 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:54.102 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:54.102 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.102 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:54.102 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.102 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:54.102 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:54.103 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:54.103 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.103 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:54.103 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.103 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:54.103 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.103 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:54.103 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:54.103 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:54.103 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.103 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:54.103 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.103 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:54.103 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:54.104 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.104 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83290/exe 2024-03-11T20:53:54.104 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo misc 2024-03-11T20:53:54.104 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.104 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:54.104 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.104 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.104 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.104 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.104 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.104 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.104 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.104 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.104 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.105 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.105 INFO:tasks.workunit.client.0.smithi129.stdout:test misc on pid 83555 2024-03-11T20:53:54.111 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=misc 2024-03-11T20:53:54.111 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83555 2024-03-11T20:53:54.111 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test misc on pid 83555' 2024-03-11T20:53:54.111 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83555 2024-03-11T20:53:54.111 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T20:53:54.111 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:54.111 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:54.112 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:54.112 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:54.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:54.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:54.112 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:54.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:54.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:54.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:54.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.112 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:54.113 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:54.113 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:54.113 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:54.113 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:54.113 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.113 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:54.113 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.113 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:54.113 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:54.113 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:54.113 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.113 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:54.113 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.113 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:54.113 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:54.114 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:54.114 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.114 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:54.114 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.114 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:54.114 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.114 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_cmd.log 2024-03-11T20:53:54.114 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ cmd: /' 2024-03-11T20:53:54.114 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_cmd 2024-03-11T20:53:54.114 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:54.114 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-03-11T20:53:54.114 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:54.114 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:54.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:54.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:54.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:54.115 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:54.115 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:54.115 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:54.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:54.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.115 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:54.115 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:54.115 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:54.116 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.116 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:54.116 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:54.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:54.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:54.116 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:54.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:54.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:54.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:54.116 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.117 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:54.117 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:54.117 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:54.117 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:54.117 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:54.117 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.117 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.117 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.117 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s pool 2024-03-11T20:53:54.117 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.117 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.117 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.117 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:54.117 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.118 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:54.118 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' pool' 2024-03-11T20:53:54.118 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:54.118 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.118 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:54.118 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.118 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:54.118 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.118 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.118 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.118 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:54.119 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.119 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:54.119 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.119 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.119 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:54.119 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.119 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.119 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.119 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.119 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.119 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.120 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.120 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.120 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.120 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:54.120 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.120 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.120 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:54.120 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.120 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:54.120 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.120 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.120 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.120 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.120 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:54.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:54.121 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.121 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:54.121 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.122 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:54.122 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.122 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.122 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:54.122 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.122 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:54.122 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.122 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:54.122 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:54.122 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo pool 2024-03-11T20:53:54.122 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83376/exe 2024-03-11T20:53:54.122 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.122 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.122 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.123 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.123 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.123 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.123 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:54.123 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.123 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.123 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.123 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.123 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.123 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.123 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.124 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.124 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.124 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.124 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.124 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.124 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.124 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.124 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.124 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.124 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.124 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.124 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.124 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.125 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.125 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:54.125 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:54.125 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:54.125 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:54.125 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.125 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:54.125 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.125 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:54.125 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:54.125 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.126 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.126 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.126 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.126 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.126 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.126 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.126 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=pool 2024-03-11T20:53:54.126 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83651 2024-03-11T20:53:54.126 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test pool on pid 83651' 2024-03-11T20:53:54.126 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83651 2024-03-11T20:53:54.127 INFO:tasks.workunit.client.0.smithi129.stdout:test pool on pid 83651 2024-03-11T20:53:54.131 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T20:53:54.131 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:54.131 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.131 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.131 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.132 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.132 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.132 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.132 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s read_operations 2024-03-11T20:53:54.132 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' read_operations' 2024-03-11T20:53:54.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_io 2024-03-11T20:53:54.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-03-11T20:53:54.132 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_io.log 2024-03-11T20:53:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83441/exe 2024-03-11T20:53:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:54.137 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.137 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:54.137 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.137 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83410/exe 2024-03-11T20:53:54.137 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ io: /' 2024-03-11T20:53:54.137 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.137 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.137 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.137 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.137 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.137 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.137 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.137 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.137 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.137 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.138 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:54.138 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.138 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.138 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.138 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.138 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.138 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.138 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.138 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:54.138 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:54.139 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:54.139 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.139 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:54.139 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.139 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:54.139 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:54.139 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:54.139 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.139 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:54.139 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.139 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:54.139 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:54.139 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:54.139 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:54.139 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ ec_io: /' 2024-03-11T20:53:54.140 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:54.140 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.140 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:54.140 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_ec_io.log 2024-03-11T20:53:54.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_ec_io 2024-03-11T20:53:54.140 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:54.140 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ list: /' 2024-03-11T20:53:54.140 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:54.140 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.140 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:54.140 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.141 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:54.141 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:54.141 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.141 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.141 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.141 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:54.141 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.141 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:54.141 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.141 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.141 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.141 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.141 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.141 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:54.141 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.142 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.142 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:54.142 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.142 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.142 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_list 2024-03-11T20:53:54.142 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.142 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.142 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.142 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.142 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.142 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.142 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.142 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.142 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.143 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.143 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.143 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.143 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.143 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.143 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.143 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.143 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.143 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.143 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.143 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.143 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.143 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_list.log 2024-03-11T20:53:54.143 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo read_operations 2024-03-11T20:53:54.144 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:54.144 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:54.144 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:54.144 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:54.144 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.144 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:54.144 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.144 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:54.144 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:54.144 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:54.144 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.144 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:54.144 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.144 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:54.145 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:54.145 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:54.145 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.145 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:54.145 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.145 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:54.145 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.145 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83454/exe 2024-03-11T20:53:54.145 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=read_operations 2024-03-11T20:53:54.145 INFO:tasks.workunit.client.0.smithi129.stdout:test read_operations on pid 83737 2024-03-11T20:53:54.148 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83737 2024-03-11T20:53:54.148 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test read_operations on pid 83737' 2024-03-11T20:53:54.148 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83737 2024-03-11T20:53:54.148 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T20:53:54.148 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:54.148 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-03-11T20:53:54.148 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:54.148 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s snapshots 2024-03-11T20:53:54.148 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:54.148 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' snapshots' 2024-03-11T20:53:54.148 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:54.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:54.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:54.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo snapshots 2024-03-11T20:53:54.149 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:54.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:54.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:54.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:54.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:54.149 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:54.149 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:54.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:54.150 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.150 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:54.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:54.150 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.150 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:54.150 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:54.150 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:54.150 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:54.151 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.151 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:54.151 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.151 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:54.151 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:54.151 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.151 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.151 INFO:tasks.workunit.client.0.smithi129.stdout:test snapshots on pid 83752 2024-03-11T20:53:54.156 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=snapshots 2024-03-11T20:53:54.156 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83752 2024-03-11T20:53:54.156 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.156 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.157 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test snapshots on pid 83752' 2024-03-11T20:53:54.157 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83752 2024-03-11T20:53:54.157 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T20:53:54.157 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:54.157 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.157 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.157 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.157 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.157 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.157 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.157 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.157 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.157 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ ec_list: /' 2024-03-11T20:53:54.158 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:54.158 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:54.158 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:54.158 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.158 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:54.158 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.158 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:54.158 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:54.158 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.158 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:54.158 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:54.159 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:54.159 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.159 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:54.159 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.159 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:54.159 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:54.159 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:54.159 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.159 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:54.159 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.159 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:54.159 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:54.160 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:54.160 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.160 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:54.160 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.160 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:54.160 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.160 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_ec_list 2024-03-11T20:53:54.160 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:54.160 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:54.160 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:54.160 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.160 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:54.160 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.160 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:54.161 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:54.161 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.161 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:54.161 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.161 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:54.161 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.161 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:54.161 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.161 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:54.161 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:54.162 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_ec_list.log 2024-03-11T20:53:54.162 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-03-11T20:53:54.162 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:54.162 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:54.162 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:54.162 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.162 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:54.162 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.162 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:54.162 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:54.162 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.162 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:54.162 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.163 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:54.163 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.163 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:54.163 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.163 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:54.163 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:54.163 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:54.163 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s watch_notify 2024-03-11T20:53:54.163 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.163 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.163 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.164 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.164 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.164 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' watch_notify' 2024-03-11T20:53:54.164 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo watch_notify 2024-03-11T20:53:54.164 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.164 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:54.164 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:54.164 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:54.164 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:54.164 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:54.164 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:54.164 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.164 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:54.165 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:54.165 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:54.165 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:54.165 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:54.165 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.165 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:54.165 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.165 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:54.165 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:54.165 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.165 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:54.165 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:54.165 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.166 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:54.166 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.166 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:54.166 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:54.166 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.166 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:54.166 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.166 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:54.166 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.166 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.166 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.166 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:54.167 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.167 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:54.167 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.167 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=watch_notify 2024-03-11T20:53:54.167 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83830 2024-03-11T20:53:54.167 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test watch_notify on pid 83830' 2024-03-11T20:53:54.167 INFO:tasks.workunit.client.0.smithi129.stdout:test watch_notify on pid 83830 2024-03-11T20:53:54.170 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83830 2024-03-11T20:53:54.170 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-11T20:53:54.170 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:54.171 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.171 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.171 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.171 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.171 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.171 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.171 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.171 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.171 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.171 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.171 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.172 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.172 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.172 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.173 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-03-11T20:53:54.173 INFO:tasks.workunit.client.0.smithi129.stderr:++ printf %25s write_operations 2024-03-11T20:53:54.173 INFO:tasks.workunit.client.0.smithi129.stderr:+ r=' write_operations' 2024-03-11T20:53:54.173 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:54.173 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:54.173 INFO:tasks.workunit.client.0.smithi129.stderr:++ awk '{print $1}' 2024-03-11T20:53:54.173 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:54.173 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:54.173 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.173 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:54.173 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:54.173 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:54.174 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:54.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:54.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:54.174 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:54.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:54.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:54.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:54.174 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.175 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:54.175 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:54.175 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:54.175 INFO:tasks.workunit.client.0.smithi129.stderr:++ echo write_operations 2024-03-11T20:53:54.175 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:54.175 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:54.175 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.175 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:54.175 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.175 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:54.175 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:54.175 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.175 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:54.175 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.176 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:54.176 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.176 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.176 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.176 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:54.176 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.176 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:54.176 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.176 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:54.176 INFO:tasks.workunit.client.0.smithi129.stdout:test write_operations on pid 83846 2024-03-11T20:53:54.184 INFO:tasks.workunit.client.0.smithi129.stderr:+ ff=write_operations 2024-03-11T20:53:54.184 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83846 2024-03-11T20:53:54.184 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo 'test write_operations on pid 83846' 2024-03-11T20:53:54.184 INFO:tasks.workunit.client.0.smithi129.stderr:+ pids[$f]=83846 2024-03-11T20:53:54.184 INFO:tasks.workunit.client.0.smithi129.stderr:+ ret=0 2024-03-11T20:53:54.184 INFO:tasks.workunit.client.0.smithi129.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-03-11T20:53:54.184 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 1 -eq 1 ']' 2024-03-11T20:53:54.184 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.184 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.185 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T20:53:54.185 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82642 2024-03-11T20:53:54.185 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82642 2024-03-11T20:53:54.185 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.185 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.185 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.185 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.185 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83555/exe 2024-03-11T20:53:54.185 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.185 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.185 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.185 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.185 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.185 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -f /etc/bashrc ']' 2024-03-11T20:53:54.186 INFO:tasks.workunit.client.0.smithi129.stderr:+ . /etc/bashrc 2024-03-11T20:53:54.186 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -z '' ']' 2024-03-11T20:53:54.186 INFO:tasks.workunit.client.0.smithi129.stderr:++ BASHRCSOURCED=Y 2024-03-11T20:53:54.186 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.186 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.186 INFO:tasks.workunit.client.0.smithi129.stderr:++ shopt -q login_shell 2024-03-11T20:53:54.186 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.186 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.186 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.186 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.186 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.186 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.187 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.187 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.187 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.187 INFO:tasks.workunit.client.0.smithi129.stderr:+++ umask 2024-03-11T20:53:54.187 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.187 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:54.187 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:54.187 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:54.187 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.187 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.187 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.187 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.187 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:54.187 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.187 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.188 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' 0022 -eq 0 ']' 2024-03-11T20:53:54.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:54.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.189 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:54.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ SHELL=/bin/bash 2024-03-11T20:53:54.189 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.189 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:54.189 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.189 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:54.189 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.189 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:54.189 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:54.190 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.190 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.190 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:54.190 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:54.190 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.190 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.190 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:54.190 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:54.190 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:54.190 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.190 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.190 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:54.190 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.190 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:54.191 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.191 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-11T20:53:54.191 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.191 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.191 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:54.191 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-11T20:53:54.191 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:54.191 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.191 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.191 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:54.191 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.191 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:54.191 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.191 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.191 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-11T20:53:54.192 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.192 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-11T20:53:54.192 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.192 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-11T20:53:54.192 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.192 INFO:tasks.workunit.client.0.smithi129.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-11T20:53:54.192 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.192 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-11T20:53:54.192 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.192 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-11T20:53:54.192 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.192 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_misc 2024-03-11T20:53:54.192 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_misc.log 2024-03-11T20:53:54.192 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83651/exe 2024-03-11T20:53:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ misc: /' 2024-03-11T20:53:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.193 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.194 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.194 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.194 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.194 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.194 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.194 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.194 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.194 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.194 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.194 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.194 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.194 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.194 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.195 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.195 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.195 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.195 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.195 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.195 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_pool 2024-03-11T20:53:54.195 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83737/exe 2024-03-11T20:53:54.195 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.195 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.195 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.196 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.196 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.196 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.196 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ pool: /' 2024-03-11T20:53:54.196 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.196 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.196 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.196 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.196 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.196 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:54.196 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:54.196 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:54.196 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.196 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:54.196 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.197 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:54.197 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:54.197 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.197 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_read_operations 2024-03-11T20:53:54.197 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ read_operations: /' 2024-03-11T20:53:54.197 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_read_operations.log 2024-03-11T20:53:54.197 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_pool.log 2024-03-11T20:53:54.197 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'grep=grep --color=auto' 2024-03-11T20:53:54.197 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-11T20:53:54.197 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-11T20:53:54.197 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.197 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-11T20:53:54.197 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.198 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorls.sh 2024-03-11T20:53:54.198 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' '!' -t 0 ']' 2024-03-11T20:53:54.198 INFO:tasks.workunit.client.0.smithi129.stderr:+++ return 2024-03-11T20:53:54.198 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.198 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-11T20:53:54.198 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.198 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-11T20:53:54.198 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export S_COLORS=auto 2024-03-11T20:53:54.198 INFO:tasks.workunit.client.0.smithi129.stderr:+++ S_COLORS=auto 2024-03-11T20:53:54.198 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.198 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-11T20:53:54.198 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.199 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-11T20:53:54.199 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.199 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:54.199 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:54.199 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:54.199 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.199 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:54.199 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.199 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:54.199 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:54.199 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.199 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:54.199 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.199 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:54.200 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.200 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:54.200 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.200 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:54.200 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:54.200 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:54.200 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:54.200 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:54.200 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.200 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:54.200 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.201 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:54.201 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:54.201 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.201 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:54.201 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-11T20:53:54.201 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.201 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.201 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-11T20:53:54.201 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-11T20:53:54.201 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.201 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.201 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.201 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-11T20:53:54.201 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.202 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.202 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-11T20:53:54.202 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-11T20:53:54.202 INFO:tasks.workunit.client.0.smithi129.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-11T20:53:54.202 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.202 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:54.202 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:54.202 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:54.202 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.202 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:54.203 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.203 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:54.203 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:54.203 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.203 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:54.203 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.203 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:54.203 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.203 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.203 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:54.203 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.203 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.204 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:54.204 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.204 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.204 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:54.204 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.204 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.204 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.204 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.204 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.204 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.204 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.205 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.205 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.205 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.205 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.205 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.205 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.205 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.205 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.205 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.205 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:54.206 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.206 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83752/exe 2024-03-11T20:53:54.206 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.206 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.206 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.206 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.206 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.206 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.206 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.206 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-11T20:53:54.206 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-11T20:53:54.206 INFO:tasks.workunit.client.0.smithi129.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-11T20:53:54.206 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.207 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-11T20:53:54.207 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.207 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-11T20:53:54.207 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-11T20:53:54.207 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.207 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-11T20:53:54.207 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.207 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/gawk.sh 2024-03-11T20:53:54.207 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.207 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-11T20:53:54.208 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.208 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/lang.sh 2024-03-11T20:53:54.208 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup 2024-03-11T20:53:54.208 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.208 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.208 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.208 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/locale 2024-03-11T20:53:54.208 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.208 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.208 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.209 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.209 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.209 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.209 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.209 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.209 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.209 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.209 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.209 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.209 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.209 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.209 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.209 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.209 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.210 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.210 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.210 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_snapshots.log 2024-03-11T20:53:54.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.210 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.210 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.210 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ snapshots: /' 2024-03-11T20:53:54.211 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83830/exe 2024-03-11T20:53:54.211 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_snapshots 2024-03-11T20:53:54.211 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.211 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.211 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-11T20:53:54.211 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.211 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-11T20:53:54.211 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-11T20:53:54.211 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.211 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.211 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.211 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.211 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.212 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.212 INFO:tasks.workunit.client.0.smithi129.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-11T20:53:54.212 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.212 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.212 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.212 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.212 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-03-11T20:53:54.212 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_watch_notify 2024-03-11T20:53:54.212 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ watch_notify: /' 2024-03-11T20:53:54.212 INFO:tasks.workunit.client.0.smithi129.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-11T20:53:54.212 INFO:tasks.workunit.client.0.smithi129.stderr:++++ export LANG=en_US.UTF-8 2024-03-11T20:53:54.213 INFO:tasks.workunit.client.0.smithi129.stderr:++++ LANG=en_US.UTF-8 2024-03-11T20:53:54.213 INFO:tasks.workunit.client.0.smithi129.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-11T20:53:54.213 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-11T20:53:54.213 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.213 INFO:tasks.workunit.client.0.smithi129.stderr:+++ LANG=en_US.UTF-8 2024-03-11T20:53:54.213 INFO:tasks.workunit.client.0.smithi129.stderr:+++ unset LANG_backup config 2024-03-11T20:53:54.213 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n '' ']' 2024-03-11T20:53:54.213 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-11T20:53:54.213 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' dumb = linux ']' 2024-03-11T20:53:54.213 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.213 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-11T20:53:54.213 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.213 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/less.sh 2024-03-11T20:53:54.214 INFO:tasks.workunit.client.0.smithi129.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-11T20:53:54.214 INFO:tasks.workunit.client.0.smithi129.stderr:++ for i in /etc/profile.d/*.sh 2024-03-11T20:53:54.214 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-11T20:53:54.214 INFO:tasks.workunit.client.0.smithi129.stderr:++ '[' '' ']' 2024-03-11T20:53:54.214 INFO:tasks.workunit.client.0.smithi129.stderr:++ . /etc/profile.d/which2.sh 2024-03-11T20:53:54.214 INFO:tasks.workunit.client.0.smithi129.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-11T20:53:54.214 INFO:tasks.workunit.client.0.smithi129.stderr:+++++ readlink /proc/83846/exe 2024-03-11T20:53:54.214 INFO:tasks.workunit.client.0.smithi129.stderr:++++ basename /usr/bin/bash 2024-03-11T20:53:54.214 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_declare='declare -f' 2024-03-11T20:53:54.214 INFO:tasks.workunit.client.0.smithi129.stderr:+++ which_opt=-f 2024-03-11T20:53:54.214 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export which_declare 2024-03-11T20:53:54.214 INFO:tasks.workunit.client.0.smithi129.stderr:+++ export -f which 2024-03-11T20:53:54.214 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset i 2024-03-11T20:53:54.215 INFO:tasks.workunit.client.0.smithi129.stderr:++ unset -f pathmunge 2024-03-11T20:53:54.215 INFO:tasks.workunit.client.0.smithi129.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-11T20:53:54.215 INFO:tasks.workunit.client.0.smithi129.stderr:+ export PATH 2024-03-11T20:53:54.215 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-11T20:53:54.215 INFO:tasks.workunit.client.0.smithi129.stderr:+ unset rc 2024-03-11T20:53:54.215 INFO:tasks.workunit.client.0.smithi129.stderr:+ sed 's/^/ write_operations: /' 2024-03-11T20:53:54.215 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph_test_neorados_write_operations 2024-03-11T20:53:54.215 INFO:tasks.workunit.client.0.smithi129.stderr:+ tee ceph_test_neorados_write_operations.log 2024-03-11T20:53:55.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:55 smithi129 ceph-mon[30184]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:55.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:55.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:53:55.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3460988003' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-11T20:53:55.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:55 smithi129 ceph-mon[42066]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:55.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:55.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:53:55.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3460988003' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-11T20:53:55.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:55 smithi157 ceph-mon[33359]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:53:55.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:53:55.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:53:55.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3460988003' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-11T20:53:56.478 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:53:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:53:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:53:59.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3745554127' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi129-83703-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:53:59.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3522262629' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi129-83754-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:53:59.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:58 smithi129 ceph-mon[30184]: osdmap e73: 8 total, 8 up, 8 in 2024-03-11T20:53:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/314482637' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi129-82567-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:53:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:53:58 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi129-83703-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:53:59.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3745554127' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi129-83703-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:53:59.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3522262629' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi129-83754-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:53:59.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:58 smithi129 ceph-mon[42066]: osdmap e73: 8 total, 8 up, 8 in 2024-03-11T20:53:59.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/314482637' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi129-82567-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:53:59.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:53:58 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi129-83703-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:53:59.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3745554127' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi129-83703-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:53:59.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3522262629' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi129-83754-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:53:59.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:58 smithi157 ceph-mon[33359]: osdmap e73: 8 total, 8 up, 8 in 2024-03-11T20:53:59.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/314482637' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi129-82567-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:53:59.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:53:58 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi129-83703-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:00.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[42066]: pgmap v61: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:00.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi129-82637-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:00.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1384198750' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi129-82913-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:00.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4237950257' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi129-82594-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:00.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3591573416' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi129-82608-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:00.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3605649432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi129-82555-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:00.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[42066]: osdmap e74: 8 total, 8 up, 8 in 2024-03-11T20:54:00.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[42066]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:00.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:00.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi129-83754-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:00.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[30184]: pgmap v61: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:00.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi129-82637-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:00.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1384198750' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi129-82913-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:00.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4237950257' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi129-82594-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:00.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3591573416' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi129-82608-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:00.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3605649432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi129-82555-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:00.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[30184]: osdmap e74: 8 total, 8 up, 8 in 2024-03-11T20:54:00.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[30184]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:00.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:00.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:00 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi129-83754-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:00.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:00 smithi157 ceph-mon[33359]: pgmap v61: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:54:00.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:00 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi129-82637-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:00.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:00 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1384198750' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi129-82913-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:00.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:00 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4237950257' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi129-82594-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:00.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:00 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3591573416' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi129-82608-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:00.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:00 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3605649432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi129-82555-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:00.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:00 smithi157 ceph-mon[33359]: osdmap e74: 8 total, 8 up, 8 in 2024-03-11T20:54:00.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:00 smithi157 ceph-mon[33359]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:00.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:00 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:00.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:00 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi129-83754-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:01.815 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-03-11T20:54:01.815 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [----------] Global test environment set-up. 2024-03-11T20:54:01.815 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [----------] 12 tests from AsioRados 2024-03-11T20:54:01.815 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-03-11T20:54:01.815 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1 ms) 2024-03-11T20:54:01.815 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-03-11T20:54:01.815 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-03-11T20:54:01.815 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-03-11T20:54:01.815 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-03-11T20:54:01.815 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-03-11T20:54:01.815 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (2 ms) 2024-03-11T20:54:01.816 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-03-11T20:54:01.816 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (1 ms) 2024-03-11T20:54:01.816 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-03-11T20:54:01.816 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (1 ms) 2024-03-11T20:54:01.816 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-03-11T20:54:01.816 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-03-11T20:54:01.816 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-03-11T20:54:01.816 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (0 ms) 2024-03-11T20:54:01.816 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-03-11T20:54:01.816 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (0 ms) 2024-03-11T20:54:01.816 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-03-11T20:54:01.816 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1 ms) 2024-03-11T20:54:01.816 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-03-11T20:54:01.816 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1 ms) 2024-03-11T20:54:01.817 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-03-11T20:54:01.817 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (1 ms) 2024-03-11T20:54:01.817 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [----------] 12 tests from AsioRados (11 ms total) 2024-03-11T20:54:01.817 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: 2024-03-11T20:54:01.817 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [----------] Global test environment tear-down 2024-03-11T20:54:01.817 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (8113 ms total) 2024-03-11T20:54:01.817 INFO:tasks.workunit.client.0.smithi129.stdout: api_asio: [ PASSED ] 12 tests. 2024-03-11T20:54:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: pgmap v63: 356 pgs: 211 creating+peering, 132 active+clean, 13 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:54:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: pgmap v64: 356 pgs: 211 creating+peering, 132 active+clean, 13 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:54:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/314482637' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi129-82567-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi129-83703-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3745554127' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi129-83703-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1817845772' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi129-82918-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/174124001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi129-83015-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2184304063' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83094-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1484664744' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi129-82672-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1634614873' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2167395915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83150-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1360292520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83279-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3166046626' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi129-83122-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2862157847' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi129-82679-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: osdmap e75: 8 total, 8 up, 8 in 2024-03-11T20:54:02.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: pgmap v63: 356 pgs: 211 creating+peering, 132 active+clean, 13 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:54:02.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: pgmap v64: 356 pgs: 211 creating+peering, 132 active+clean, 13 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:54:02.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/314482637' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi129-82567-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:02.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi129-83703-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:02.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3745554127' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi129-83703-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:02.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:02.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1817845772' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi129-82918-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/174124001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi129-83015-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2184304063' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83094-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1484664744' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi129-82672-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1634614873' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2167395915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83150-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1360292520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83279-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3166046626' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi129-83122-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2862157847' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi129-82679-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: osdmap e75: 8 total, 8 up, 8 in 2024-03-11T20:54:02.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1008704522' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi129-83754-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:02.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3016685796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi129-83342-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3522262629' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi129-83754-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:02.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: osdmap e76: 8 total, 8 up, 8 in 2024-03-11T20:54:02.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3335346089' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-11T20:54:02.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2305105740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi129-82818-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1008704522' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi129-83754-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:02.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3016685796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi129-83342-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3522262629' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi129-83754-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:02.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: osdmap e76: 8 total, 8 up, 8 in 2024-03-11T20:54:02.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3335346089' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-11T20:54:02.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2305105740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi129-82818-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: pgmap v63: 356 pgs: 211 creating+peering, 132 active+clean, 13 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:54:02.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: pgmap v64: 356 pgs: 211 creating+peering, 132 active+clean, 13 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:54:02.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/314482637' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi129-82567-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:02.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi129-83703-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:02.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3745554127' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi129-83703-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:02.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1817845772' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi129-82918-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/174124001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi129-83015-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2184304063' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83094-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1484664744' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi129-82672-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1634614873' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2167395915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83150-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1360292520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83279-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3166046626' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi129-83122-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2862157847' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi129-82679-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: osdmap e75: 8 total, 8 up, 8 in 2024-03-11T20:54:02.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1008704522' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi129-83754-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:02.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3016685796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi129-83342-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:02.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3522262629' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi129-83754-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:02.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: osdmap e76: 8 total, 8 up, 8 in 2024-03-11T20:54:02.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3335346089' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-11T20:54:02.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2305105740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi129-82818-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.467 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-03-11T20:54:03.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] Global test environment set-up. 2024-03-11T20:54:03.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 7 tests from LibRadosList 2024-03-11T20:54:03.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-03-11T20:54:03.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.ListObjects (431 ms) 2024-03-11T20:54:03.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-03-11T20:54:03.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (31 ms) 2024-03-11T20:54:03.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-03-11T20:54:03.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo1,foo2,foo3 2024-03-11T20:54:03.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo1 2024-03-11T20:54:03.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo2 2024-03-11T20:54:03.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo3 2024-03-11T20:54:03.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo1,foo4,foo5 2024-03-11T20:54:03.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo4 2024-03-11T20:54:03.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo5 2024-03-11T20:54:03.468 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo1 2024-03-11T20:54:03.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo6,foo7 2024-03-11T20:54:03.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo7 2024-03-11T20:54:03.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo6 2024-03-11T20:54:03.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-11T20:54:03.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo4 2024-03-11T20:54:03.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo5 2024-03-11T20:54:03.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns2:foo7 2024-03-11T20:54:03.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns2:foo6 2024-03-11T20:54:03.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo1 2024-03-11T20:54:03.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo1 2024-03-11T20:54:03.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo2 2024-03-11T20:54:03.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo3 2024-03-11T20:54:03.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (89 ms) 2024-03-11T20:54:03.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-03-11T20:54:03.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 1 0 2024-03-11T20:54:03.469 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 10 0 2024-03-11T20:54:03.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 13 0 2024-03-11T20:54:03.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 7 0 2024-03-11T20:54:03.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 14 0 2024-03-11T20:54:03.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 0 0 2024-03-11T20:54:03.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 15 0 2024-03-11T20:54:03.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 11 0 2024-03-11T20:54:03.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 5 0 2024-03-11T20:54:03.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 8 0 2024-03-11T20:54:03.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 6 0 2024-03-11T20:54:03.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 3 0 2024-03-11T20:54:03.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 4 0 2024-03-11T20:54:03.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 12 0 2024-03-11T20:54:03.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 9 0 2024-03-11T20:54:03.470 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 2 0 2024-03-11T20:54:03.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-11T20:54:03.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (64 ms) 2024-03-11T20:54:03.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-03-11T20:54:03.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: x cursor=MIN 2024-03-11T20:54:03.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=1 cursor=10:02547ec2:::1:head 2024-03-11T20:54:03.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=10 cursor=10:52ea6a34:::10:head 2024-03-11T20:54:03.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=13 cursor=10:566253c9:::13:head 2024-03-11T20:54:03.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=7 cursor=10:5c6b0b28:::7:head 2024-03-11T20:54:03.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=14 cursor=10:62a1935d:::14:head 2024-03-11T20:54:03.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=0 cursor=10:6cac518f:::0:head 2024-03-11T20:54:03.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=15 cursor=10:863748b0:::15:head 2024-03-11T20:54:03.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=11 cursor=10:89d3ae78:::11:head 2024-03-11T20:54:03.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=5 cursor=10:b29083e3:::5:head 2024-03-11T20:54:03.471 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=8 cursor=10:bd63b0f1:::8:head 2024-03-11T20:54:03.472 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=6 cursor=10:c4fdafeb:::6:head 2024-03-11T20:54:03.472 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=3 cursor=10:cfc208b3:::3:head 2024-03-11T20:54:03.472 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=4 cursor=10:d83876eb:::4:head 2024-03-11T20:54:03.472 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=12 cursor=10:de5d7c5f:::12:head 2024-03-11T20:54:03.472 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=9 cursor=10:e960b815:::9:head 2024-03-11T20:54:03.472 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > oid=2 cursor=10:f905c69b:::2:head 2024-03-11T20:54:03.472 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: FIRST> seek to MIN oid=1 2024-03-11T20:54:03.472 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=1 cursor=10:02547ec2:::1:head 2024-03-11T20:54:03.472 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:02547ec2:::1:head 2024-03-11T20:54:03.472 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:02547ec2:::1:head -> 1 2024-03-11T20:54:03.472 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=10 cursor=10:52ea6a34:::10:head 2024-03-11T20:54:03.472 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:52ea6a34:::10:head 2024-03-11T20:54:03.472 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:52ea6a34:::10:head -> 10 2024-03-11T20:54:03.472 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=13 cursor=10:566253c9:::13:head 2024-03-11T20:54:03.472 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:566253c9:::13:head 2024-03-11T20:54:03.473 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:566253c9:::13:head -> 13 2024-03-11T20:54:03.473 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=7 cursor=10:5c6b0b28:::7:head 2024-03-11T20:54:03.473 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:5c6b0b28:::7:head 2024-03-11T20:54:03.473 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:5c6b0b28:::7:head -> 7 2024-03-11T20:54:03.524 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=14 cursor=10:62a1935d:::14:head 2024-03-11T20:54:03.524 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:62a1935d:::14:head 2024-03-11T20:54:03.524 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:62a1935d:::14:head -> 14 2024-03-11T20:54:03.524 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=0 cursor=10:6cac518f:::0:head 2024-03-11T20:54:03.524 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:6cac518f:::0:head 2024-03-11T20:54:03.524 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:6cac518f:::0:head -> 0 2024-03-11T20:54:03.524 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=15 cursor=10:863748b0:::15:head 2024-03-11T20:54:03.524 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:863748b0:::15:head 2024-03-11T20:54:03.524 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:863748b0:::15:head -> 15 2024-03-11T20:54:03.524 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=11 cursor=10:89d3ae78:::11:head 2024-03-11T20:54:03.525 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:89d3ae78:::11:head 2024-03-11T20:54:03.525 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:89d3ae78:::11:head -> 11 2024-03-11T20:54:03.525 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=5 cursor=10:b29083e3:::5:head 2024-03-11T20:54:03.525 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:b29083e3:::5:head 2024-03-11T20:54:03.525 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:b29083e3:::5:head -> 5 2024-03-11T20:54:03.525 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=8 cursor=10:bd63b0f1:::8:head 2024-03-11T20:54:03.525 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:bd63b0f1:::8:head 2024-03-11T20:54:03.525 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:bd63b0f1:::8:head -> 8 2024-03-11T20:54:03.525 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=6 cursor=10:c4fdafeb:::6:head 2024-03-11T20:54:03.525 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:c4fdafeb:::6:head 2024-03-11T20:54:03.525 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:c4fdafeb:::6:head -> 6 2024-03-11T20:54:03.525 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=3 cursor=10:cfc208b3:::3:head 2024-03-11T20:54:03.525 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:cfc208b3:::3:head 2024-03-11T20:54:03.526 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:cfc208b3:::3:head -> 3 2024-03-11T20:54:03.526 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=4 cursor=10:d83876eb:::4:head 2024-03-11T20:54:03.526 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:d83876eb:::4:head 2024-03-11T20:54:03.526 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:d83876eb:::4:head -> 4 2024-03-11T20:54:03.526 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=12 cursor=10:de5d7c5f:::12:head 2024-03-11T20:54:03.526 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:de5d7c5f:::12:head 2024-03-11T20:54:03.526 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:de5d7c5f:::12:head -> 12 2024-03-11T20:54:03.526 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=9 cursor=10:e960b815:::9:head 2024-03-11T20:54:03.526 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:e960b815:::9:head 2024-03-11T20:54:03.526 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:e960b815:::9:head -> 9 2024-03-11T20:54:03.526 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : oid=2 cursor=10:f905c69b:::2:head 2024-03-11T20:54:03.526 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:f905c69b:::2:head 2024-03-11T20:54:03.526 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:f905c69b:::2:head -> 2 2024-03-11T20:54:03.526 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:cfc208b3:::3:head 2024-03-11T20:54:03.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=10:cfc208b3:::3:head expected=10:cfc208b3:::3:head 2024-03-11T20:54:03.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:cfc208b3:::3:head -> 3 2024-03-11T20:54:03.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=3 expected=3 2024-03-11T20:54:03.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:e960b815:::9:head 2024-03-11T20:54:03.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=10:e960b815:::9:head expected=10:e960b815:::9:head 2024-03-11T20:54:03.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:e960b815:::9:head -> 9 2024-03-11T20:54:03.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=9 expected=9 2024-03-11T20:54:03.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:bd63b0f1:::8:head 2024-03-11T20:54:03.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=10:bd63b0f1:::8:head expected=10:bd63b0f1:::8:head 2024-03-11T20:54:03.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:bd63b0f1:::8:head -> 8 2024-03-11T20:54:03.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=8 expected=8 2024-03-11T20:54:03.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:f905c69b:::2:head 2024-03-11T20:54:03.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=10:f905c69b:::2:head expected=10:f905c69b:::2:head 2024-03-11T20:54:03.527 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:f905c69b:::2:head -> 2 2024-03-11T20:54:03.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=2 expected=2 2024-03-11T20:54:03.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:de5d7c5f:::12:head 2024-03-11T20:54:03.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=10:de5d7c5f:::12:head expected=10:de5d7c5f:::12:head 2024-03-11T20:54:03.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:de5d7c5f:::12:head -> 12 2024-03-11T20:54:03.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=12 expected=12 2024-03-11T20:54:03.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:d83876eb:::4:head 2024-03-11T20:54:03.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=10:d83876eb:::4:head expected=10:d83876eb:::4:head 2024-03-11T20:54:03.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:d83876eb:::4:head -> 4 2024-03-11T20:54:03.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=4 expected=4 2024-03-11T20:54:03.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:c4fdafeb:::6:head 2024-03-11T20:54:03.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=10:c4fdafeb:::6:head expected=10:c4fdafeb:::6:head 2024-03-11T20:54:03.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:c4fdafeb:::6:head -> 6 2024-03-11T20:54:03.528 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=6 expected=6 2024-03-11T20:54:03.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:b29083e3:::5:head 2024-03-11T20:54:03.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=10:b29083e3:::5:head expected=10:b29083e3:::5:head 2024-03-11T20:54:03.529 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:b29083e3:::5:head -> 5 2024-03-11T20:54:03.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: pgmap v66: 804 pgs: 211 creating+peering, 132 active+clean, 461 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 795 B/s rd, 0 op/s 2024-03-11T20:54:03.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi129-82637-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi129-82913-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi129-83703-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:03.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83094-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83150-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1008704522' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:03.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2305105740' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi129-82818-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:03.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3648425588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi129-82945-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: osdmap e77: 8 total, 8 up, 8 in 2024-03-11T20:54:03.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:54:03.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83279-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi129-82594-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi129-83122-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi129-82608-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:03.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi129-82555-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:03.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi129-83754-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:03.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: pgmap v66: 804 pgs: 211 creating+peering, 132 active+clean, 461 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 795 B/s rd, 0 op/s 2024-03-11T20:54:03.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi129-82637-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi129-82913-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi129-83703-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:03.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83094-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83150-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1008704522' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:03.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2305105740' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi129-82818-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:03.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3648425588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi129-82945-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: osdmap e77: 8 total, 8 up, 8 in 2024-03-11T20:54:03.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:54:03.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83279-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi129-82594-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi129-83122-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi129-82608-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:03.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi129-82555-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:03.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi129-83754-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:03.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi129-82918-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi129-83015-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:03.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi129-82672-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi129-82945-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi129-82679-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi129-83342-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:03.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi129-82918-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi129-83015-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:03.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi129-82672-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi129-82945-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi129-82679-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi129-83342-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:03.735 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:54:03.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: pgmap v66: 804 pgs: 211 creating+peering, 132 active+clean, 461 unknown; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 795 B/s rd, 0 op/s 2024-03-11T20:54:03.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi129-82637-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi129-82913-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi129-83703-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:03.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83094-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83150-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1008704522' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:03.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2305105740' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi129-82818-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:03.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3648425588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi129-82945-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: osdmap e77: 8 total, 8 up, 8 in 2024-03-11T20:54:03.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:54:03.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83279-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi129-82594-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi129-83122-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi129-82608-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:03.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi129-82555-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:03.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi129-83754-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:03.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi129-82918-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi129-83015-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:03.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi129-82672-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi129-82945-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi129-82679-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi129-83342-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:03.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:03.984 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=5 expecte list_parallel: process_1_[83474]: starting. 2024-03-11T20:54:03.984 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[83474]: creating pool ceph_test_rados_list_parallel.smithi129-83365 2024-03-11T20:54:03.984 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[83474]: created object 0... 2024-03-11T20:54:03.984 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[83474]: created object 25... 2024-03-11T20:54:03.984 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[83474]: created object 49... 2024-03-11T20:54:03.984 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[83474]: finishing. 2024-03-11T20:54:03.984 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_1_[83474]: shutting down. 2024-03-11T20:54:03.985 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[83479]: starting. 2024-03-11T20:54:03.985 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[83479]: listing objects. 2024-03-11T20:54:03.985 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[83479]: listed object 0... 2024-03-11T20:54:03.985 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[83479]: listed object 25... 2024-03-11T20:54:03.985 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[83479]: saw 50 objects 2024-03-11T20:54:03.985 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_2_[83479]: shutting down. 2024-03-11T20:54:03.985 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:03.985 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[84507]: starting. 2024-03-11T20:54:03.985 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[84507]: creating pool ceph_test_rados_list_parallel.smithi129-83365 2024-03-11T20:54:03.985 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[84507]: created object 0... 2024-03-11T20:54:03.985 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[84507]: created object 25... 2024-03-11T20:54:03.985 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[84507]: created object 49... 2024-03-11T20:54:03.985 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[84507]: finishing. 2024-03-11T20:54:03.985 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_3_[84507]: shutting down. 2024-03-11T20:54:03.986 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:03.986 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[84508]: starting. 2024-03-11T20:54:03.986 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[84508]: listing objects. 2024-03-11T20:54:03.986 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[84508]: listed object 0... 2024-03-11T20:54:03.986 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[84508]: listed object 25... 2024-03-11T20:54:03.986 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[84508]: saw 43 objects 2024-03-11T20:54:03.986 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_4_[84508]: shutting down. 2024-03-11T20:54:03.986 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:03.986 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[84509]: starting. 2024-03-11T20:54:03.986 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[84509]: removed 25 objects... 2024-03-11T20:54:03.986 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[84509]: removed half of the objects 2024-03-11T20:54:03.986 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[84509]: removed 50 objects... 2024-03-11T20:54:03.986 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[84509]: removed 50 objects 2024-03-11T20:54:03.986 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_5_[84509]: shutting down. 2024-03-11T20:54:03.987 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:03.987 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:03.987 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[84585]: starting. 2024-03-11T20:54:03.987 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[84585]: creating pool ceph_test_rados_list_parallel.smithi129-83365 2024-03-11T20:54:03.987 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[84585]: created object 0... 2024-03-11T20:54:03.987 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[84585]: created object 25... 2024-03-11T20:54:03.987 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[84585]: created object 49... 2024-03-11T20:54:03.987 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[84585]: finishing. 2024-03-11T20:54:03.987 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_6_[84585]: shutting down. 2024-03-11T20:54:03.987 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:03.987 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:03.987 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[84586]: starting. 2024-03-11T20:54:03.987 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[84586]: listing objects. 2024-03-11T20:54:03.987 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[84586]: listed object 0... 2024-03-11T20:54:03.987 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[84586]: listed object 25... 2024-03-11T20:54:03.988 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[84586]: saw 50 objects 2024-03-11T20:54:03.988 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_7_[84586]: shutting down. 2024-03-11T20:54:03.988 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:03.988 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:03.988 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[84587]: starting. 2024-03-11T20:54:03.988 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[84587]: added 25 objects... 2024-03-11T20:54:03.988 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[84587]: added half of the objects 2024-03-11T20:54:03.988 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[84587]: added 50 objects... 2024-03-11T20:54:03.988 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[84587]: added 50 objects 2024-03-11T20:54:03.988 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_8_[84587]: shutting down. 2024-03-11T20:54:03.989 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:03.989 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:03.989 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:03.989 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[84635]: starting. 2024-03-11T20:54:04.521 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[84635]: creating pool ceph_test_rados_list_parallel.smithi129-83365 2024-03-11T20:54:04.521 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[84635]: created object 0... 2024-03-11T20:54:04.522 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[84635]: created object 25... 2024-03-11T20:54:04.522 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[84635]: created object 49... 2024-03-11T20:54:04.522 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[84635]: finishing. 2024-03-11T20:54:04.522 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_9_[84635]: shutting down. 2024-03-11T20:54:04.522 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.522 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.522 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.522 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[84636]: starting. 2024-03-11T20:54:04.522 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[84636]: listing objects. 2024-03-11T20:54:04.522 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[84636]: listed object 0... 2024-03-11T20:54:04.522 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[84636]: listed object 25... 2024-03-11T20:54:04.522 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[84636]: listed object 50... 2024-03-11T20:54:04.522 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[84636]: listed object 75... 2024-03-11T20:54:04.522 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[84636]: listed object 100... 2024-03-11T20:54:04.523 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[84636]: saw 105 objects 2024-03-11T20:54:04.523 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_10_[84636]: shutting down. 2024-03-11T20:54:04.523 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.523 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.523 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.523 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[84638]: starting. 2024-03-11T20:54:04.523 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[84638]: added 25 objects... 2024-03-11T20:54:04.523 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[84638]: added half of the objects 2024-03-11T20:54:04.523 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[84638]: added 50 objects... 2024-03-11T20:54:04.523 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[84638]: added 50 objects 2024-03-11T20:54:04.523 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_12_[84638]: shutting down. 2024-03-11T20:54:04.523 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.523 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.524 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.524 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[84637]: starting. 2024-03-11T20:54:04.524 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[84637]: added 25 objects... 2024-03-11T20:54:04.524 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[84637]: added half of the objects 2024-03-11T20:54:04.524 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[84637]: added 50 objects... 2024-03-11T20:54:04.524 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[84637]: added 50 objects 2024-03-11T20:54:04.524 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_11_[84637]: shutting down. 2024-03-11T20:54:04.524 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.524 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.524 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.524 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[84639]: starting. 2024-03-11T20:54:04.524 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[84639]: removed 25 objects... 2024-03-11T20:54:04.524 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[84639]: removed half of the objects 2024-03-11T20:54:04.524 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[84639]: removed 50 objects... 2024-03-11T20:54:04.525 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[84639]: removed 50 objects 2024-03-11T20:54:04.525 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_13_[84639]: shutting down. 2024-03-11T20:54:04.525 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.525 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.525 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.525 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.525 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[84849]: starting. 2024-03-11T20:54:04.525 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[84849]: creating pool ceph_test_rados_list_parallel.smithi129-83365 2024-03-11T20:54:04.525 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[84849]: created object 0... 2024-03-11T20:54:04.525 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[84849]: created object 25... 2024-03-11T20:54:04.525 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[84849]: created object 49... 2024-03-11T20:54:04.525 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[84849]: finishing. 2024-03-11T20:54:04.525 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_14_[84849]: shutting down. 2024-03-11T20:54:04.525 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.525 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.526 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.526 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:04.526 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[84850]: starting. 2024-03-11T20:54:04.526 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[84850]: listing objects. 2024-03-11T20:54:04.526 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[84850]: listed object 0... 2024-03-11T20:54:04.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: pgmap v69: 1156 pgs: 160 unknown, 851 creating+peering, 145 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:54:04.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi129-82637-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi129-82913-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83094-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83150-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83279-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi129-82594-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi129-83122-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi129-82608-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi129-82555-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi129-82918-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi129-83015-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi129-82672-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi129-82945-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi129-82679-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi129-83342-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi129-83101-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:04.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:04.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3599201142' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-11T20:54:04.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: osdmap e78: 8 total, 8 up, 8 in 2024-03-11T20:54:04.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4024199307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:04.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:04.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi129-83101-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:04.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi129-83703-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi129-83703-1"}]': finished 2024-03-11T20:54:04.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi129-83754-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi129-83754-1"}]': finished 2024-03-11T20:54:04.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: osdmap e79: 8 total, 8 up, 8 in 2024-03-11T20:54:04.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: Health check update: 30 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:04.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:04.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: pgmap v69: 1156 pgs: 160 unknown, 851 creating+peering, 145 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:54:04.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi129-82637-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi129-82913-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83094-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83150-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83279-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi129-82594-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi129-83122-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi129-82608-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi129-82555-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi129-82918-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi129-83015-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi129-82672-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi129-82945-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi129-82679-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi129-83342-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:04.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi129-83101-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:04.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:04.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3599201142' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-11T20:54:04.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: osdmap e78: 8 total, 8 up, 8 in 2024-03-11T20:54:04.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4024199307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:04.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:04.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi129-83101-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:04.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi129-83703-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi129-83703-1"}]': finished 2024-03-11T20:54:04.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi129-83754-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi129-83754-1"}]': finished 2024-03-11T20:54:04.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: osdmap e79: 8 total, 8 up, 8 in 2024-03-11T20:54:04.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: Health check update: 30 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:04.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:04.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: pgmap v69: 1156 pgs: 160 unknown, 851 creating+peering, 145 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T20:54:04.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi129-82637-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi129-82913-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83094-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83150-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83279-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi129-82594-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi129-83122-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi129-82608-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi129-82555-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi129-82918-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi129-83015-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi129-82672-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi129-82945-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi129-82679-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi129-83342-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:04.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:04.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi129-83101-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:04.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:04.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3599201142' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-11T20:54:04.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: osdmap e78: 8 total, 8 up, 8 in 2024-03-11T20:54:04.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4024199307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:04.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:04.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi129-83101-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:04.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi129-83703-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi129-83703-1"}]': finished 2024-03-11T20:54:04.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi129-83754-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi129-83754-1"}]': finished 2024-03-11T20:54:04.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: osdmap e79: 8 total, 8 up, 8 in 2024-03-11T20:54:04.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: Health check update: 30 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:04.740 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:05.000 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[84850]: listed objec api_c_read_operations: Running main() from gmock_main.cc 2024-03-11T20:54:05.000 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-03-11T20:54:05.000 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-03-11T20:54:05.000 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-03-11T20:54:05.000 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-03-11T20:54:05.000 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-03-11T20:54:05.000 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-03-11T20:54:05.000 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (373 ms) 2024-03-11T20:54:05.000 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-03-11T20:54:05.000 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (11 ms) 2024-03-11T20:54:05.000 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-03-11T20:54:05.001 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (9 ms) 2024-03-11T20:54:05.001 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-03-11T20:54:05.001 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (32 ms) 2024-03-11T20:54:05.001 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-03-11T20:54:05.001 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (10 ms) 2024-03-11T20:54:05.001 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-03-11T20:54:05.001 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (8 ms) 2024-03-11T20:54:05.001 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-03-11T20:54:05.001 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (5 ms) 2024-03-11T20:54:05.001 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-03-11T20:54:05.001 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (6 ms) 2024-03-11T20:54:05.001 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-03-11T20:54:05.001 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (6 ms) 2024-03-11T20:54:05.001 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-03-11T20:54:05.002 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (6 ms) 2024-03-11T20:54:05.002 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-03-11T20:54:05.002 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (5 ms) 2024-03-11T20:54:05.002 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-03-11T20:54:05.002 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (7 ms) 2024-03-11T20:54:05.002 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-03-11T20:54:05.002 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (15 ms) 2024-03-11T20:54:05.002 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-03-11T20:54:05.002 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (10 ms) 2024-03-11T20:54:05.002 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-03-11T20:54:05.002 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (7 ms) 2024-03-11T20:54:05.002 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-03-11T20:54:05.002 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (4 ms) 2024-03-11T20:54:05.002 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (515 ms total) 2024-03-11T20:54:05.003 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: 2024-03-11T20:54:05.003 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-03-11T20:54:05.003 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (11071 ms total) 2024-03-11T20:54:05.003 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-03-11T20:54:05.005 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: Running main() from gmock_main.cc 2024-03-11T20:54:05.005 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-03-11T20:54:05.005 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [----------] Global test environment set-up. 2024-03-11T20:54:05.005 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-03-11T20:54:05.005 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-03-11T20:54:05.005 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (10979 ms) 2024-03-11T20:54:05.006 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [----------] 1 test from neocls_handler_error (10979 ms total) 2024-03-11T20:54:05.006 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: 2024-03-11T20:54:05.006 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [----------] Global test environment tear-down 2024-03-11T20:54:05.006 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [==========] 1 test from 1 test suite ran. (10979 ms total) 2024-03-11T20:54:05.006 INFO:tasks.workunit.client.0.smithi129.stdout: handler_error: [ PASSED ] 1 test. 2024-03-11T20:54:05.017 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-03-11T20:54:05.017 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-03-11T20:54:05.017 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-03-11T20:54:05.017 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-03-11T20:54:05.017 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (11062 ms) 2024-03-11T20:54:05.018 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (11062 ms total) 2024-03-11T20:54:05.018 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: 2024-03-11T20:54:05.018 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-03-11T20:54:05.018 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (11062 ms total) 2024-03-11T20:54:05.018 INFO:tasks.workunit.client.0.smithi129.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-03-11T20:54:05.019 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-03-11T20:54:05.019 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-03-11T20:54:05.019 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-03-11T20:54:05.019 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-03-11T20:54:05.019 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-03-11T20:54:05.019 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (94 ms) 2024-03-11T20:54:05.019 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-03-11T20:54:05.019 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (81 ms) 2024-03-11T20:54:05.019 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-03-11T20:54:05.019 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (10978 ms) 2024-03-11T20:54:05.020 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (11153 ms total) 2024-03-11T20:54:05.020 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: 2024-03-11T20:54:05.020 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-03-11T20:54:05.020 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (11153 ms total) 2024-03-11T20:54:05.020 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-03-11T20:54:05.047 INFO:tasks.workunit.client.0.smithi129.stdout:t 25... 2024-03-11T20:54:05.047 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[84850]: listed object 50... 2024-03-11T20:54:05.047 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[84850]: listed object 75... 2024-03-11T20:54:05.047 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[84850]: listed object 100... 2024-03-11T20:54:05.047 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[84850]: listed object 125... 2024-03-11T20:54:05.047 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[84850]: saw 150 objects 2024-03-11T20:54:05.048 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_15_[84850]: shutting down. 2024-03-11T20:54:05.048 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:05.048 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:05.048 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:05.048 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:05.048 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[84851]: starting. 2024-03-11T20:54:05.048 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[84851]: added 25 objects... 2024-03-11T20:54:05.048 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[84851]: added half of the objects 2024-03-11T20:54:05.048 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[84851]: added 50 objects... 2024-03-11T20:54:05.048 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[84851]: added 50 objects 2024-03-11T20:54:05.048 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: process_16_[84851]: shutting down. 2024-03-11T20:54:05.048 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:05.048 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:05.048 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:05.049 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:05.049 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******************************* 2024-03-11T20:54:05.049 INFO:tasks.workunit.client.0.smithi129.stdout: list_parallel: ******* SUCCESS ********** 2024-03-11T20:54:05.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: Running main() from gmock_main.cc 2024-03-11T20:54:05.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-03-11T20:54:05.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [----------] Global test environment set-up. 2024-03-11T20:54:05.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-03-11T20:54:05.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-03-11T20:54:05.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (69 ms) 2024-03-11T20:54:05.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-03-11T20:54:05.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (87 ms) 2024-03-11T20:54:05.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-03-11T20:54:05.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (11000 ms) 2024-03-11T20:54:05.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-03-11T20:54:05.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.760803+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi129-82637-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:05.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.760905+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi129-82913-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:05.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.760991+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83094-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:05.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.761070+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83150-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:05.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.761145+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83279-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:05.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.761218+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi129-82594-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:05.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.761293+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi129-83122-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:05.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.761369+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi129-82608-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:05.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.761442+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi129-82555-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:05.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.761514+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi129-82918-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:05.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.761599+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi129-83015-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:05.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.761674+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:05.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.761750+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi129-82672-1","app": "rados","yes_i_really_mean_ api_service: [==========] Running 4 tests from 1 test suite. 2024-03-11T20:54:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [----------] Global test environment set-up. 2024-03-11T20:54:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [----------] 4 tests from LibRadosService 2024-03-11T20:54:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-03-11T20:54:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5059 ms) 2024-03-11T20:54:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-03-11T20:54:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ OK ] LibRadosService.RegisterLate (27 ms) 2024-03-11T20:54:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-03-11T20:54:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: cluster: 2024-03-11T20:54:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: id: 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:54:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: health: HEALTH_WARN 2024-03-11T20:54:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 7 pool(s) do not have an application enabled 2024-03-11T20:54:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-03-11T20:54:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: services: 2024-03-11T20:54:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-11T20:54:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-11T20:54:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-11T20:54:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-11T20:54:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-11T20:54:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-03-11T20:54:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: data: 2024-03-11T20:54:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pools: 13 pools, 356 pgs 2024-03-11T20:54:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: objects: 198 objects, 582 KiB 2024-03-11T20:54:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: usage: 231 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:54:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pgs: 3.652% pgs unknown 2024-03-11T20:54:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 59.270% pgs not active 2024-03-11T20:54:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 211 creating+peering 2024-03-11T20:54:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 132 active+clean 2024-03-11T20:54:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 13 unknown 2024-03-11T20:54:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-03-11T20:54:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: io: 2024-03-11T20:54:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-11T20:54:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-03-11T20:54:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-03-11T20:54:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: cluster: 2024-03-11T20:54:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: id: 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:54:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: health: HEALTH_WARN 2024-03-11T20:54:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 20 pool(s) do not have an application enabled 2024-03-11T20:54:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-03-11T20:54:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: services: 2024-03-11T20:54:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-11T20:54:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-11T20:54:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-11T20:54:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: foo: 13 portals active (1 hosts, 3 zones) 2024-03-11T20:54:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-11T20:54:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-03-11T20:54:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: data: 2024-03-11T20:54:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pools: 27 pools, 804 pgs 2024-03-11T20:54:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: objects: 198 objects, 582 KiB 2024-03-11T20:54:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: usage: 231 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:54:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pgs: 57.338% pgs unknown 2024-03-11T20:54:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 26.244% pgs not active 2024-03-11T20:54:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 461 unknown 2024-03-11T20:54:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 211 creating+peering 2024-03-11T20:54:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 132 active+clean 2024-03-11T20:54:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-03-11T20:54:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: io: 2024-03-11T20:54:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: client: 795 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-11T20:54:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-03-11T20:54:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-03-11T20:54:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: cluster: 2024-03-11T20:54:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: id: 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T20:54:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: health: HEALTH_WARN 2024-03-11T20:54:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 30 pool(s) do not have an application enabled 2024-03-11T20:54:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-03-11T20:54:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: services: 2024-03-11T20:54:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-11T20:54:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-11T20:54:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-11T20:54:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-03-11T20:54:06.478 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:54:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:54:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:54:06.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:06.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:06 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi129-83101-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]': finished 2024-03-11T20:54:06.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:06.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:06 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi129-83101-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]': finished 2024-03-11T20:54:06.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:06.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:06 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi129-83101-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]': finished 2024-03-11T20:54:07.785 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: launit": true}]': finished 2024-03-11T20:54:07.785 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.761831+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi129-82945-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:07.785 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.761908+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:07.785 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.761980+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi129-82679-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:07.785 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.762053+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi129-83342-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:07.785 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.762125+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:07.785 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:02.773080+0000 mon.b [INF] from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi129-83101-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:07.785 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:03.418169+0000 mon.c [INF] from='client.? 172.21.15.129:0/4024199307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:07.785 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:03.717654+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi129-83101-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:07.785 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:03.768479+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi129-83703-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi129-83703-1"}]': finished 2024-03-11T20:54:07.786 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:03.769150+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi129-83754-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi129-83754-1"}]': finished 2024-03-11T20:54:07.786 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:04.080553+0000 mon.a [WRN] Health check update: 30 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:07.786 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:04.135399+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:07.786 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:04.986601+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi129-83101-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]': finished 2024-03-11T20:54:07.786 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:04.986704+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:07.786 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:04.999337+0000 mon.b [INF] from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:07.786 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:05.000858+0000 mon.c [INF] from='client.? 172.21.15.129:0/3745554127' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:07.786 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:05.067206+0000 mon.c [INF] from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:07.786 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:05.078084+0000 mon.c [INF] from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:07.786 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:05.103990+0000 mon.c [INF] from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:07.786 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:05.108293+0000 mon.b [INF] from='client.? 172.21.15.129:0/450167167' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-11T20:54:07.786 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:05.630059+0000 mon.b [INF] from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.0"}]: dispatch 2024-03-11T20:54:07.787 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:05.631020+0000 mon.b [INF] from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.1"}]: dispatch 2024-03-11T20:54:07.787 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:05.631475+0000 mon.b [INF] from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.2"}]: dispatch 2024-03-11T20:54:07.787 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:05.631893+0000 mon.b [INF] from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.3"}]: dispatch 2024-03-11T20:54:07.787 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:05.632305+0000 mon.b [INF] from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.4"}]: dispatch 2024-03-11T20:54:07.787 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:05.632747+0000 mon.b [INF] from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.5"}]: dispatch 2024-03-11T20:54:07.787 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:05.633205+0000 mon.b [INF] from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.6"}]: dispatch 2024-03-11T20:54:07.788 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:05.633650+0000 mon.b [INF] from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.7"}]: dispatch 2024-03-11T20:54:07.788 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:05.634051+0000 mon.b [INF] from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.8"}]: dispatch 2024-03-11T20:54:07.788 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:05.634462+0000 mon.b [INF] from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.9"}]: dispatch 2024-03-11T20:54:07.788 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:06.553525+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:07.788 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:06.566135+0000 mon.b [INF] from='client.? 172.21.15.129:0/669401043' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi129-82555-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: pgmap v72: 1076 pgs: 240 unknown, 697 creating+peering, 139 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:54:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3745554127' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/450167167' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-11T20:54:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2532669377' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-11T20:54:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.0"}]: dispatch 2024-03-11T20:54:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.0"}]: dispatch 2024-03-11T20:54:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.1"}]: dispatch 2024-03-11T20:54:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.1"}]: dispatch 2024-03-11T20:54:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.2"}]: dispatch 2024-03-11T20:54:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.2"}]: dispatch 2024-03-11T20:54:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.3"}]: dispatch 2024-03-11T20:54:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.3"}]: dispatch 2024-03-11T20:54:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.4"}]: dispatch 2024-03-11T20:54:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.4"}]: dispatch 2024-03-11T20:54:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.5"}]: dispatch 2024-03-11T20:54:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.5"}]: dispatch 2024-03-11T20:54:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.6"}]: dispatch 2024-03-11T20:54:08.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: pgmap v72: 1076 pgs: 240 unknown, 697 creating+peering, 139 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:54:08.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:08.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:08.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3745554127' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:08.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:08.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:08.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:08.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/450167167' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-11T20:54:08.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2532669377' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-11T20:54:08.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.0"}]: dispatch 2024-03-11T20:54:08.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.0"}]: dispatch 2024-03-11T20:54:08.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.1"}]: dispatch 2024-03-11T20:54:08.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.1"}]: dispatch 2024-03-11T20:54:08.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.2"}]: dispatch 2024-03-11T20:54:08.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.2"}]: dispatch 2024-03-11T20:54:08.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.3"}]: dispatch 2024-03-11T20:54:08.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.3"}]: dispatch 2024-03-11T20:54:08.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.4"}]: dispatch 2024-03-11T20:54:08.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.4"}]: dispatch 2024-03-11T20:54:08.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:08.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.5"}]: dispatch 2024-03-11T20:54:08.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.5"}]: dispatch 2024-03-11T20:54:08.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.6"}]: dispatch 2024-03-11T20:54:08.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.6"}]: dispatch 2024-03-11T20:54:08.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.7"}]: dispatch 2024-03-11T20:54:08.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.7"}]: dispatch 2024-03-11T20:54:08.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.8"}]: dispatch 2024-03-11T20:54:08.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.8"}]: dispatch 2024-03-11T20:54:08.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.9"}]: dispatch 2024-03-11T20:54:08.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.9"}]: dispatch 2024-03-11T20:54:08.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: osdmap e80: 8 total, 8 up, 8 in 2024-03-11T20:54:08.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:08.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: osdmap e81: 8 total, 8 up, 8 in 2024-03-11T20:54:08.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/669401043' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi129-82555-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:08.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/714930713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:08.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/414450280' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:08.234 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:08.234 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:08.234 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:08.234 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:08.234 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:08.234 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:08.234 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:08.234 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.6"}]: dispatch 2024-03-11T20:54:08.234 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.7"}]: dispatch 2024-03-11T20:54:08.234 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.7"}]: dispatch 2024-03-11T20:54:08.235 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.8"}]: dispatch 2024-03-11T20:54:08.235 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.8"}]: dispatch 2024-03-11T20:54:08.235 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.9"}]: dispatch 2024-03-11T20:54:08.235 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.9"}]: dispatch 2024-03-11T20:54:08.235 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: osdmap e80: 8 total, 8 up, 8 in 2024-03-11T20:54:08.235 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:08.235 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: osdmap e81: 8 total, 8 up, 8 in 2024-03-11T20:54:08.235 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/669401043' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi129-82555-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:08.235 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/714930713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:08.235 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/414450280' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:08.235 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:08.235 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:08.235 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:08.236 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:08.236 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:08.236 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:08.236 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:08.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: pgmap v72: 1076 pgs: 240 unknown, 697 creating+peering, 139 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:54:08.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:08.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3745554127' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/450167167' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-11T20:54:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2532669377' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-11T20:54:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.0"}]: dispatch 2024-03-11T20:54:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.0"}]: dispatch 2024-03-11T20:54:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.1"}]: dispatch 2024-03-11T20:54:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.1"}]: dispatch 2024-03-11T20:54:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.2"}]: dispatch 2024-03-11T20:54:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.2"}]: dispatch 2024-03-11T20:54:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.3"}]: dispatch 2024-03-11T20:54:08.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.3"}]: dispatch 2024-03-11T20:54:08.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.4"}]: dispatch 2024-03-11T20:54:08.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.4"}]: dispatch 2024-03-11T20:54:08.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:08.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.5"}]: dispatch 2024-03-11T20:54:08.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.5"}]: dispatch 2024-03-11T20:54:08.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.6"}]: dispatch 2024-03-11T20:54:08.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.6"}]: dispatch 2024-03-11T20:54:08.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.7"}]: dispatch 2024-03-11T20:54:08.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.7"}]: dispatch 2024-03-11T20:54:08.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.8"}]: dispatch 2024-03-11T20:54:08.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.8"}]: dispatch 2024-03-11T20:54:08.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.9"}]: dispatch 2024-03-11T20:54:08.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.9"}]: dispatch 2024-03-11T20:54:08.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: osdmap e80: 8 total, 8 up, 8 in 2024-03-11T20:54:08.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:08.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: osdmap e81: 8 total, 8 up, 8 in 2024-03-11T20:54:08.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/669401043' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi129-82555-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:08.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/714930713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:08.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/414450280' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:08.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:08.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:08.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:08.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:08.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:08.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:08.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:09.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:06.566458+0000 mon.b [INF] from='client.? 172.21.15.129:0/714930713' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.034 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:06.577592+0000 mon.b [INF] from='client.? 172.21.15.129:0/414450280' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:06.820546+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:06.823675+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:09.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:06.824533+0000 mon.c [INF] from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:09.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:07.066463+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:09.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:07.067126+0000 mon.c [INF] from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:09.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:07.303670+0000 mon.a [INF] from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:09.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:05.109062+0000 client.admin [INF] onexx 2024-03-11T20:54:09.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:07.779141+0000 mon.a [INF] from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:09.035 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:07.783829+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:09.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:07.785260+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi129-83703-1"}]': finished 2024-03-11T20:54:09.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:07.785342+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:09.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:07.798507+0000 mon.c [INF] from='client.? 172.21.15.129:0/3745554127' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:09.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:07.798882+0000 mon.c [INF] from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:09.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:07.799827+0000 mon.c [INF] from='client.? 172.21.15.129:0/4024199307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.036 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:08.053934+0000 mon.a [INF] from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi129-82594-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:09.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: onexx 2024-03-11T20:54:09.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.7 deep-scrub starts 2024-03-11T20:54:09.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.7 deep-scrub ok 2024-03-11T20:54:09.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.1 deep-scrub starts 2024-03-11T20:54:09.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.1 deep-scrub ok 2024-03-11T20:54:09.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.3 deep-scrub starts 2024-03-11T20:54:09.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.6 deep-scrub starts 2024-03-11T20:54:09.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.6 deep-scrub ok 2024-03-11T20:54:09.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.3 deep-scrub ok 2024-03-11T20:54:09.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.4 deep-scrub starts 2024-03-11T20:54:09.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.4 deep-scrub ok 2024-03-11T20:54:09.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.8 deep-scrub starts 2024-03-11T20:54:09.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.8 deep-scrub ok 2024-03-11T20:54:09.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.0 deep-scrub starts 2024-03-11T20:54:09.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.0 deep-scrub ok 2024-03-11T20:54:09.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: pgmap v75: 852 pgs: 296 creating+peering, 556 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 564 op/s 2024-03-11T20:54:09.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.5 deep-scrub starts 2024-03-11T20:54:09.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.5 deep-scrub ok 2024-03-11T20:54:09.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.9 deep-scrub starts 2024-03-11T20:54:09.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: 23.9 deep-scrub ok 2024-03-11T20:54:09.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:09.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:09.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:09.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi129-83703-1"}]': finished 2024-03-11T20:54:09.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:09.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3745554127' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:09.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:09.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4024199307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: osdmap e82: 8 total, 8 up, 8 in 2024-03-11T20:54:09.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi129-82594-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:09.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-11T20:54:09.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi129-82555-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:09.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:09.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:09.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:09.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:09.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:09.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.483 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi129-82918-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:09.483 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:09.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: onexx 2024-03-11T20:54:09.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.7 deep-scrub starts 2024-03-11T20:54:09.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.7 deep-scrub ok 2024-03-11T20:54:09.484 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.1 deep-scrub starts 2024-03-11T20:54:09.484 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.1 deep-scrub ok 2024-03-11T20:54:09.484 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.3 deep-scrub starts 2024-03-11T20:54:09.484 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.6 deep-scrub starts 2024-03-11T20:54:09.484 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.6 deep-scrub ok 2024-03-11T20:54:09.484 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.3 deep-scrub ok 2024-03-11T20:54:09.484 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.4 deep-scrub starts 2024-03-11T20:54:09.484 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.4 deep-scrub ok 2024-03-11T20:54:09.484 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.8 deep-scrub starts 2024-03-11T20:54:09.484 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.8 deep-scrub ok 2024-03-11T20:54:09.484 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.0 deep-scrub starts 2024-03-11T20:54:09.484 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.0 deep-scrub ok 2024-03-11T20:54:09.484 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: pgmap v75: 852 pgs: 296 creating+peering, 556 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 564 op/s 2024-03-11T20:54:09.485 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.5 deep-scrub starts 2024-03-11T20:54:09.485 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.5 deep-scrub ok 2024-03-11T20:54:09.485 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.9 deep-scrub starts 2024-03-11T20:54:09.485 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: 23.9 deep-scrub ok 2024-03-11T20:54:09.485 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:09.485 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:09.485 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:09.485 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi129-83703-1"}]': finished 2024-03-11T20:54:09.485 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:09.485 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3745554127' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:09.485 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:09.485 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4024199307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.485 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: osdmap e82: 8 total, 8 up, 8 in 2024-03-11T20:54:09.486 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi129-82594-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:09.486 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-11T20:54:09.486 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi129-82555-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.486 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.486 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:09.486 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.486 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:09.486 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:09.486 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:09.486 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:09.486 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:09.486 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.486 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi129-82918-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:09.487 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:09.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: onexx 2024-03-11T20:54:09.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.7 deep-scrub starts 2024-03-11T20:54:09.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.7 deep-scrub ok 2024-03-11T20:54:09.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.1 deep-scrub starts 2024-03-11T20:54:09.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.1 deep-scrub ok 2024-03-11T20:54:09.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.3 deep-scrub starts 2024-03-11T20:54:09.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.6 deep-scrub starts 2024-03-11T20:54:09.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.6 deep-scrub ok 2024-03-11T20:54:09.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.3 deep-scrub ok 2024-03-11T20:54:09.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.4 deep-scrub starts 2024-03-11T20:54:09.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.4 deep-scrub ok 2024-03-11T20:54:09.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.8 deep-scrub starts 2024-03-11T20:54:09.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.8 deep-scrub ok 2024-03-11T20:54:09.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.0 deep-scrub starts 2024-03-11T20:54:09.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.0 deep-scrub ok 2024-03-11T20:54:09.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: pgmap v75: 852 pgs: 296 creating+peering, 556 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 564 op/s 2024-03-11T20:54:09.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.5 deep-scrub starts 2024-03-11T20:54:09.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.5 deep-scrub ok 2024-03-11T20:54:09.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.9 deep-scrub starts 2024-03-11T20:54:09.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: 23.9 deep-scrub ok 2024-03-11T20:54:09.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:09.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:09.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:09.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi129-83703-1"}]': finished 2024-03-11T20:54:09.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:09.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3745554127' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:09.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:09.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4024199307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: osdmap e82: 8 total, 8 up, 8 in 2024-03-11T20:54:09.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi129-82594-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:09.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-11T20:54:09.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi129-82555-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:09.490 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.490 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:09.490 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:09.490 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:09.490 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:09.490 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:09.490 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:09.490 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi129-82918-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:09.490 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:10.333 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:08.054995+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}] delete_pools_parallel: process_1_[83455]: starting. 2024-03-11T20:54:10.333 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[83455]: creating pool ceph_test_rados_delete_pools_parallel.smithi129-83361 2024-03-11T20:54:10.333 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[83455]: created object 0... 2024-03-11T20:54:10.333 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[83455]: created object 25... 2024-03-11T20:54:10.333 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[83455]: created object 49... 2024-03-11T20:54:10.333 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[83455]: finishing. 2024-03-11T20:54:10.334 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_1_[83455]: shutting down. 2024-03-11T20:54:10.334 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_2_[83457]: starting. 2024-03-11T20:54:10.334 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_2_[83457]: deleting pool ceph_test_rados_delete_pools_parallel.smithi129-83361 2024-03-11T20:54:10.334 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_2_[83457]: shutting down. 2024-03-11T20:54:10.334 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******************************* 2024-03-11T20:54:10.334 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[84752]: starting. 2024-03-11T20:54:10.334 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[84752]: creating pool ceph_test_rados_delete_pools_parallel.smithi129-83361 2024-03-11T20:54:10.334 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[84752]: created object 0... 2024-03-11T20:54:10.334 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[84752]: created object 25... 2024-03-11T20:54:10.334 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[84752]: created object 49... 2024-03-11T20:54:10.334 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[84752]: finishing. 2024-03-11T20:54:10.334 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_3_[84752]: shutting down. 2024-03-11T20:54:10.335 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******************************* 2024-03-11T20:54:10.335 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[84757]: starting. 2024-03-11T20:54:10.335 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[84757]: listing objects. 2024-03-11T20:54:10.335 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[84757]: listed object 0... 2024-03-11T20:54:10.335 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[84757]: listed object 25... 2024-03-11T20:54:10.335 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[84757]: saw 50 objects 2024-03-11T20:54:10.335 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_5_[84757]: shutting down. 2024-03-11T20:54:10.335 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******************************* 2024-03-11T20:54:10.335 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_4_[84753]: starting. 2024-03-11T20:54:10.336 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_4_[84753]: deleting pool ceph_test_rados_delete_pools_parallel.smithi129-83361 2024-03-11T20:54:10.336 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: process_4_[84753]: shutting down. 2024-03-11T20:54:10.336 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******************************* 2024-03-11T20:54:10.336 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******************************* 2024-03-11T20:54:10.336 INFO:tasks.workunit.client.0.smithi129.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-03-11T20:54:10.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: 23.2 deep-scrub starts 2024-03-11T20:54:10.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: 23.2 deep-scrub ok 2024-03-11T20:54:10.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: pgmap v77: 948 pgs: 160 unknown, 264 creating+peering, 524 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 9.6 MiB/s rd, 30 MiB/s wr, 453 op/s 2024-03-11T20:54:10.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi129-82918-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi129-82594-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:10.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi129-82555-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:10.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:10.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:10.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi129-83703-1"}]': finished 2024-03-11T20:54:10.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:10.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-11T20:54:10.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: osdmap e83: 8 total, 8 up, 8 in 2024-03-11T20:54:10.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/450167167' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-11T20:54:10.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:10.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3522262629' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:10.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4024199307' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi129-82567-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-11T20:54:10.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: twoxx 2024-03-11T20:54:10.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:10.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi129-82594-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:10.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi129-82567-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-11T20:54:10.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-11T20:54:10.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:10.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: Health check update: 17 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:10.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi129-83703-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:10.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:10.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:10.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi129-82672-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:10.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:10.732 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: 23.2 deep-scrub starts 2024-03-11T20:54:10.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: 23.2 deep-scrub ok 2024-03-11T20:54:10.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: pgmap v77: 948 pgs: 160 unknown, 264 creating+peering, 524 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 9.6 MiB/s rd, 30 MiB/s wr, 453 op/s 2024-03-11T20:54:10.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi129-82918-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi129-82594-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:10.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi129-82555-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:10.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:10.733 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:10.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi129-83703-1"}]': finished 2024-03-11T20:54:10.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:10.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-11T20:54:10.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: osdmap e83: 8 total, 8 up, 8 in 2024-03-11T20:54:10.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/450167167' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-11T20:54:10.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:10.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3522262629' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:10.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4024199307' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi129-82567-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-11T20:54:10.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: twoxx 2024-03-11T20:54:10.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:10.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi129-82594-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:10.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi129-82567-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-11T20:54:10.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-11T20:54:10.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:10.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: Health check update: 17 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:10.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi129-83703-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:10.736 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:10.736 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:10.736 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi129-82672-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.736 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:10.736 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.736 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:10.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: 23.2 deep-scrub starts 2024-03-11T20:54:10.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: 23.2 deep-scrub ok 2024-03-11T20:54:10.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: pgmap v77: 948 pgs: 160 unknown, 264 creating+peering, 524 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 9.6 MiB/s rd, 30 MiB/s wr, 453 op/s 2024-03-11T20:54:10.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi129-82918-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi129-82594-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:10.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi129-82555-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:10.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:10.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:10.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi129-83703-1"}]': finished 2024-03-11T20:54:10.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:10.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-11T20:54:10.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: osdmap e83: 8 total, 8 up, 8 in 2024-03-11T20:54:10.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/450167167' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-11T20:54:10.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:10.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3522262629' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:10.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4024199307' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi129-82567-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-11T20:54:10.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: twoxx 2024-03-11T20:54:10.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:10.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi129-82594-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:10.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi129-82567-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-11T20:54:10.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-11T20:54:10.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:10.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: Health check update: 17 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:10.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi129-83703-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:10.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:10.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:10.740 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi129-82672-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.740 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:10.740 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:10.740 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:12.339 INFO:tasks.workunit.client.0.smithi129.stdout:: dispatch 2024-03-11T20:54:12.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:08.302699+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi129-82555-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:12.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:08.302926+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:12.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:08.304611+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:12.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:08.304765+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:12.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:08.305085+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:12.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:08.305129+0000 mon.c [INF] from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:12.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:08.305369+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi129-83703-1"}]: dispatch 2024-03-11T20:54:12.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:08.305634+0000 mon.c [INF] from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:12.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:08.305702+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:12.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:08.305968+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:12.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:08.306114+0000 mon.c [INF] from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi129-82918-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:12.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.025008+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:12.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.030805+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:12.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.031136+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi129-82918-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:12.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.033905+0000 mon.a [INF] from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi129-82594-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.034018+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi129-82555-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:12.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.034121+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:12.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.034201+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:12.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.034261+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi129-83703-1"}]': finished 2024-03-11T20:54:12.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.034329+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi129-82567-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:12.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.035358+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-11T20:54:12.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.045650+0000 mon.b [INF] from='client.? 172.21.15.129:0/450167167' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-11T20:54:12.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.049692+0000 mon.b [INF] from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:12.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.051101+0000 mon.c [INF] from='client.? 172.21.15.129:0/3522262629' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:12.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.051977+0000 mon.c [INF] from='client.? 172.21.15.129:0/4024199307' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi129-82567-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-11T20:54:12.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.054118+0000 client.admin [INF] twoxx 2024-03-11T20:54:12.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.063040+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:12.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.063357+0000 mon.a [INF] from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi129-82594-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:12.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.063789+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi129-82567-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-11T20:54:12.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.064996+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-11T20:54:12.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.069352+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:12.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.086755+0000 mon.a [WRN] Health check update: 17 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:12.569 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.625613+0000 mon.a [INF] from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi129-83703-2", "profile": [ "k=2", "m=1", "crush-failure-domain= cls: Running main() from gmock_main.cc 2024-03-11T20:54:12.569 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-03-11T20:54:12.569 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [----------] Global test environment set-up. 2024-03-11T20:54:12.569 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-03-11T20:54:12.570 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-03-11T20:54:12.570 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [ OK ] NeoRadosCls.DNE (11018 ms) 2024-03-11T20:54:12.570 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-03-11T20:54:12.570 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (7569 ms) 2024-03-11T20:54:12.570 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [----------] 2 tests from NeoRadosCls (18587 ms total) 2024-03-11T20:54:12.570 INFO:tasks.workunit.client.0.smithi129.stdout: cls: 2024-03-11T20:54:12.571 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [----------] Global test environment tear-down 2024-03-11T20:54:12.571 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [==========] 2 tests from 1 test suite ran. (18587 ms total) 2024-03-11T20:54:12.571 INFO:tasks.workunit.client.0.smithi129.stdout: cls: [ PASSED ] 2 tests. 2024-03-11T20:54:12.572 INFO:tasks.workunit.client.0.smithi129.stdout:osd"]}]: dispatch 2024-03-11T20:54:12.572 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.626452+0000 mon.a [INF] from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:12.573 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.626783+0000 mon.a [INF] from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:12.573 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.627104+0000 mon.a [INF] from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:12.573 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.627586+0000 mon.a [INF] from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi129-82672-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:12.573 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.628275+0000 mon.a [INF] from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:12.573 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:09.628755+0000 mon.a [INF] from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:12.573 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.038035+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.573 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.038136+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.038215+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi129-82918-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.038298+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi129-83754-1"}]': finished 2024-03-11T20:54:12.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.038374+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi129-82567-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-11T20:54:12.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.038453+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:12.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.038525+0000 mon.a [INF] from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi129-83703-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.574 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.038581+0000 mon.a [INF] from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi129-82672-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.038649+0000 mon.a [INF] from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.044137+0000 mon.b [INF] from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:12.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.047705+0000 mon.c [INF] from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi129-83015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:12.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.048029+0000 mon.c [INF] from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi129-82608-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:12.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.048303+0000 mon.c [INF] from='client.? 172.21.15.129:0/3522262629' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:12.575 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.049400+0000 mon.c [INF] from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi129-82918-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:12.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.327622+0000 mon.b [INF] from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:12.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.329281+0000 mon.a [INF] from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi129-83703-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi129-83703-2"}]: dispatch 2024-03-11T20:54:12.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.329863+0000 mon.a [INF] from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi129-82679-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:12.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.330116+0000 mon.a [INF] from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi129-82672-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:12.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.330365+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:12.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.330556+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi129-83015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:12.576 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.331109+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi129-82608-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:12.577 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.331495+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:12.577 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03- open_pools_parallel: process_1_[83385]: starting. 2024-03-11T20:54:12.577 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[83385]: creating pool ceph_test_rados_open_pools_parallel.smithi129-83353 2024-03-11T20:54:12.578 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[83385]: created object 0... 2024-03-11T20:54:12.578 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[83385]: created object 25... 2024-03-11T20:54:12.578 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[83385]: created object 49... 2024-03-11T20:54:12.578 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[83385]: finishing. 2024-03-11T20:54:12.578 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_1_[83385]: shutting down. 2024-03-11T20:54:12.578 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_2_[83388]: starting. 2024-03-11T20:54:12.579 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_2_[83388]: rados_pool_create. 2024-03-11T20:54:12.579 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_2_[83388]: rados_ioctx_create. 2024-03-11T20:54:12.579 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_2_[83388]: shutting down. 2024-03-11T20:54:12.579 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: ******************************* 2024-03-11T20:54:12.579 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[84924]: starting. 2024-03-11T20:54:12.579 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[84924]: creating pool ceph_test_rados_open_pools_parallel.smithi129-83353 2024-03-11T20:54:12.580 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[84924]: created object 0... 2024-03-11T20:54:12.580 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[84924]: created object 25... 2024-03-11T20:54:12.580 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[84924]: created object 49... 2024-03-11T20:54:12.580 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[84924]: finishing. 2024-03-11T20:54:12.580 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_3_[84924]: shutting down. 2024-03-11T20:54:12.580 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: ******************************* 2024-03-11T20:54:12.580 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_4_[84925]: starting. 2024-03-11T20:54:12.580 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_4_[84925]: rados_pool_create. 2024-03-11T20:54:12.581 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_4_[84925]: rados_ioctx_create. 2024-03-11T20:54:12.581 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: process_4_[84925]: shutting down. 2024-03-11T20:54:12.581 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: ******************************* 2024-03-11T20:54:12.581 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: ******************************* 2024-03-11T20:54:12.581 INFO:tasks.workunit.client.0.smithi129.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-03-11T20:54:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi129-82918-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi129-83754-1"}]': finished 2024-03-11T20:54:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi129-82567-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-11T20:54:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi129-83703-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi129-82672-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi129-83015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:12.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi129-82608-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:12.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3522262629' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:12.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi129-82918-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:12.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: osdmap e84: 8 total, 8 up, 8 in 2024-03-11T20:54:12.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:12.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi129-82918-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi129-83754-1"}]': finished 2024-03-11T20:54:12.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi129-82567-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-11T20:54:12.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:12.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi129-83703-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi129-82672-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:12.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi129-83015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:12.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi129-82608-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:12.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3522262629' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:12.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi129-82918-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:12.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: osdmap e84: 8 total, 8 up, 8 in 2024-03-11T20:54:12.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:12.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi129-83703-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi129-83703-2"}]: dispatch 2024-03-11T20:54:12.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi129-82679-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:12.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi129-82672-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:12.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:12.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi129-83015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:12.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi129-82608-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:12.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:12.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi129-82918-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:12.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-11T20:54:12.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:12.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi129-82594-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi129-82594-16"}]': finished 2024-03-11T20:54:12.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi129-83703-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi129-83703-2"}]: dispatch 2024-03-11T20:54:12.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi129-82679-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:12.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi129-82672-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:12.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:12.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi129-83015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:12.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi129-82608-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:12.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:12.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi129-82918-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:12.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-11T20:54:12.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:12.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi129-82594-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi129-82594-16"}]': finished 2024-03-11T20:54:12.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi129-82918-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi129-83754-1"}]': finished 2024-03-11T20:54:12.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi129-82567-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-11T20:54:12.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:12.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi129-83703-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi129-82672-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:12.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:12.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi129-83015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:12.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi129-82608-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:12.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3522262629' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:12.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi129-82918-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:12.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: osdmap e84: 8 total, 8 up, 8 in 2024-03-11T20:54:12.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:12.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi129-83703-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi129-83703-2"}]: dispatch 2024-03-11T20:54:12.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi129-82679-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:12.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi129-82672-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:12.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:12.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi129-83015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:12.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi129-82608-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:12.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi129-83754-1"}]: dispatch 2024-03-11T20:54:12.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi129-82918-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:12.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-11T20:54:12.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:12.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi129-82594-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi129-82594-16"}]': finished 2024-03-11T20:54:13.586 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:54:13.588 INFO:tasks.workunit.client.0.smithi129.stdout:11T20:54:10.333634+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi129-82918-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:13.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:10.335175+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-11T20:54:13.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:11.086210+0000 mon.a [INF] from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi129-82594-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi129-82594-16"}]': finished 2024-03-11T20:54:13.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:11.086293+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]': finished 2024-03-11T20:54:13.588 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:11.086360+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi129-83754-1"}]': finished 2024-03-11T20:54:13.589 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:11.090754+0000 mon.b [INF] from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-4", "overlaypool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:13.589 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:11.122374+0000 mon.b [INF] from='client.? 172.21.15.129:0/867247583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi129-83754-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:13.589 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:12.344050+0000 mon.b [INF] from='client.? 172.21.15.129:0/450167167' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-11T20:54:13.589 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:12.344206+0000 client.admin [INF] threexx 2024-03-11T20:54:13.589 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:12.559627+0000 mon.c [INF] from='client.? 172.21.15.129:0/2200346640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:13.589 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:12.573284+0000 mon.a [INF] from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi129-83703-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi129-83703-2"}]': finished 2024-03-11T20:54:13.589 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:12.573418+0000 mon.a [INF] from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi129-82679-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi129-82679-10"}]': finished 2024-03-11T20:54:13.590 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:12.573486+0000 mon.a [INF] from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi129-82672-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi129-82672-10"}]': finished 2024-03-11T20:54:13.590 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:12.573582+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi129-83015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi129-83015-7"}]': finished 2024-03-11T20:54:13.618 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:12.573667+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi129-82608-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofi cmd: Running main() from gmock_main.cc 2024-03-11T20:54:13.618 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-03-11T20:54:13.618 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [----------] Global test environment set-up. 2024-03-11T20:54:13.618 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-03-11T20:54:13.618 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-03-11T20:54:13.618 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (9385 ms) 2024-03-11T20:54:13.618 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-03-11T20:54:13.618 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (5636 ms) 2024-03-11T20:54:13.618 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-03-11T20:54:13.618 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (4566 ms) 2024-03-11T20:54:13.619 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [----------] 3 tests from NeoRadosCmd (19587 ms total) 2024-03-11T20:54:13.619 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: 2024-03-11T20:54:13.619 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [----------] Global test environment tear-down 2024-03-11T20:54:13.619 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [==========] 3 tests from 1 test suite ran. (19587 ms total) 2024-03-11T20:54:13.619 INFO:tasks.workunit.client.0.smithi129.stdout: cmd: [ PASSED ] 3 tests. 2024-03-11T20:54:13.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: Running main() from gmock_main.cc 2024-03-11T20:54:13.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-03-11T20:54:13.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] Global test environment set-up. 2024-03-11T20:54:13.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-03-11T20:54:13.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: seed 82608 2024-03-11T20:54:13.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-03-11T20:54:13.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-03-11T20:54:13.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-03-11T20:54:13.763 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (378 ms) 2024-03-11T20:54:13.763 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-03-11T20:54:13.763 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (21 ms) 2024-03-11T20:54:13.763 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-03-11T20:54:13.763 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (8 ms) 2024-03-11T20:54:13.763 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-03-11T20:54:13.763 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (6 ms) 2024-03-11T20:54:13.763 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-03-11T20:54:13.763 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (6 ms) 2024-03-11T20:54:13.763 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-03-11T20:54:13.763 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (2 ms) 2024-03-11T20:54:13.763 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-03-11T20:54:13.763 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (4 ms) 2024-03-11T20:54:13.764 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-03-11T20:54:13.764 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (6 ms) 2024-03-11T20:54:13.764 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-03-11T20:54:13.764 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (6 ms) 2024-03-11T20:54:13.764 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-03-11T20:54:13.764 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (3 ms) 2024-03-11T20:54:13.764 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-03-11T20:54:13.764 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (4 ms) 2024-03-11T20:54:13.765 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-03-11T20:54:13.765 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (5 ms) 2024-03-11T20:54:13.765 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-03-11T20:54:13.765 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (3 ms) 2024-03-11T20:54:13.765 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-03-11T20:54:13.765 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (5 ms) 2024-03-11T20:54:13.765 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-03-11T20:54:13.765 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (14 ms) 2024-03-11T20:54:13.765 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-11T20:54:13.765 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (6 ms) 2024-03-11T20:54:13.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-03-11T20:54:13.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (5 ms) 2024-03-11T20:54:13.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-03-11T20:54:13.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (3 ms) 2024-03-11T20:54:13.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-03-11T20:54:13.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (4 ms) 2024-03-11T20:54:13.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (490 ms total) 2024-03-11T20:54:13.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: 2024-03-11T20:54:13.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-03-11T20:54:13.766 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-03-11T20:54:13.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1144 ms) 2024-03-11T20:54:13.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-03-11T20:54:13.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (13 ms) 2024-03-11T20:54:13.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-03-11T20:54:13.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (4 ms) 2024-03-11T20:54:13.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-03-11T20:54:13.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (6 ms) 2024-03-11T20:54:13.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-03-11T20:54:13.767 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (2 ms) 2024-03-11T20:54:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: pgmap v80: 836 pgs: 352 unknown, 32 creating+peering, 452 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 36 MiB/s wr, 521 op/s 2024-03-11T20:54:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]': finished 2024-03-11T20:54:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi129-83754-1"}]': finished 2024-03-11T20:54:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-4", "overlaypool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/867247583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi129-83754-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/450167167' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-11T20:54:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: threexx 2024-03-11T20:54:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: osdmap e85: 8 total, 8 up, 8 in 2024-03-11T20:54:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2200346640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi129-83703-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi129-83703-2"}]': finished 2024-03-11T20:54:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi129-82679-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi129-82679-10"}]': finished 2024-03-11T20:54:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi129-82672-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi129-82672-10"}]': finished 2024-03-11T20:54:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi129-83015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi129-83015-7"}]': finished 2024-03-11T20:54:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi129-82608-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi129-82608-22"}]': finished 2024-03-11T20:54:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi129-82918-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi129-82918-7"}]': finished 2024-03-11T20:54:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: osdmap e86: 8 total, 8 up, 8 in 2024-03-11T20:54:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-4", "overlaypool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi129-83754-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-11T20:54:13.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: pgmap v80: 836 pgs: 352 unknown, 32 creating+peering, 452 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 36 MiB/s wr, 521 op/s 2024-03-11T20:54:13.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]': finished 2024-03-11T20:54:13.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi129-83754-1"}]': finished 2024-03-11T20:54:13.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-4", "overlaypool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:13.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/867247583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi129-83754-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:13.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:13.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/450167167' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-11T20:54:13.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: threexx 2024-03-11T20:54:13.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: osdmap e85: 8 total, 8 up, 8 in 2024-03-11T20:54:13.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2200346640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:13.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi129-83703-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi129-83703-2"}]': finished 2024-03-11T20:54:13.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi129-82679-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi129-82679-10"}]': finished 2024-03-11T20:54:13.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi129-82672-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi129-82672-10"}]': finished 2024-03-11T20:54:13.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi129-83015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi129-83015-7"}]': finished 2024-03-11T20:54:13.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi129-82608-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi129-82608-22"}]': finished 2024-03-11T20:54:13.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi129-82918-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi129-82918-7"}]': finished 2024-03-11T20:54:13.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: osdmap e86: 8 total, 8 up, 8 in 2024-03-11T20:54:13.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:13.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:13.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:13.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-4", "overlaypool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:13.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi129-83754-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:13.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-11T20:54:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: pgmap v80: 836 pgs: 352 unknown, 32 creating+peering, 452 active+clean; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 36 MiB/s wr, 521 op/s 2024-03-11T20:54:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]': finished 2024-03-11T20:54:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi129-83754-1"}]': finished 2024-03-11T20:54:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-4", "overlaypool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/867247583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi129-83754-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/450167167' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-11T20:54:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: threexx 2024-03-11T20:54:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: osdmap e85: 8 total, 8 up, 8 in 2024-03-11T20:54:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2200346640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi129-83703-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi129-83703-2"}]': finished 2024-03-11T20:54:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi129-82679-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi129-82679-10"}]': finished 2024-03-11T20:54:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi129-82672-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi129-82672-10"}]': finished 2024-03-11T20:54:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi129-83015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi129-83015-7"}]': finished 2024-03-11T20:54:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi129-82608-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi129-82608-22"}]': finished 2024-03-11T20:54:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi129-82918-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi129-82918-7"}]': finished 2024-03-11T20:54:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: osdmap e86: 8 total, 8 up, 8 in 2024-03-11T20:54:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-4", "overlaypool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:13.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi129-83754-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:13.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-11T20:54:14.164 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] watch_notify: Running main() from gmock_main.cc 2024-03-11T20:54:14.165 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-03-11T20:54:14.165 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [----------] Global test environment set-up. 2024-03-11T20:54:14.165 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-03-11T20:54:14.165 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-03-11T20:54:14.165 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: handle_notify cookie 94268472983232 notify_id 335007449089 notifier_gid 24842 2024-03-11T20:54:14.165 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (10880 ms) 2024-03-11T20:54:14.165 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-03-11T20:54:14.165 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: Trying... 2024-03-11T20:54:14.165 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: handle_notify cookie 94268485679760 notify_id 356482285568 notifier_gid 25078 2024-03-11T20:54:14.165 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: Waiting for 3.000000000s 2024-03-11T20:54:14.165 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: Timed out. 2024-03-11T20:54:14.165 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: Flushing... 2024-03-11T20:54:14.165 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: Flushed... 2024-03-11T20:54:14.165 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (9165 ms) 2024-03-11T20:54:14.166 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (20045 ms total) 2024-03-11T20:54:14.166 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: 2024-03-11T20:54:14.166 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [----------] Global test environment tear-down 2024-03-11T20:54:14.166 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (20045 ms total) 2024-03-11T20:54:14.166 INFO:tasks.workunit.client.0.smithi129.stdout: watch_notify: [ PASSED ] 2 tests. 2024-03-11T20:54:14.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: pgmap v83: 860 pgs: 90 creating+peering, 62 unknown, 708 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:54:14.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:14.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:14.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]': finished 2024-03-11T20:54:14.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-4", "overlaypool": "test-rados-api-smithi129-82818-4-cache"}]': finished 2024-03-11T20:54:14.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi129-83754-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:14.983 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-11T20:54:14.984 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-4-cache", "mode": "writeback"}]: dispatch 2024-03-11T20:54:14.984 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:14.984 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/450167167' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-11T20:54:14.984 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: osdmap e87: 8 total, 8 up, 8 in 2024-03-11T20:54:14.984 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/867247583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi129-83754-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:14.984 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: fourxx 2024-03-11T20:54:14.984 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2433590356' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi129-82555-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:14.984 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2387178343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:14.984 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:14.984 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-4-cache", "mode": "writeback"}]: dispatch 2024-03-11T20:54:14.984 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:14.984 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-11T20:54:14.985 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi129-83754-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:14.985 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi129-82555-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:14.985 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:14.985 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:14.985 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: Health check update: 18 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:14.985 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: pool 'PoolQuotaPP_smithi129-82567-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-11T20:54:14.985 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:54:14.985 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-11T20:54:14.985 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:14.986 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: pgmap v83: 860 pgs: 90 creating+peering, 62 unknown, 708 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:54:14.986 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:14.986 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:14.986 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]': finished 2024-03-11T20:54:14.986 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-4", "overlaypool": "test-rados-api-smithi129-82818-4-cache"}]': finished 2024-03-11T20:54:14.986 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi129-83754-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:14.986 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-11T20:54:14.986 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-4-cache", "mode": "writeback"}]: dispatch 2024-03-11T20:54:14.986 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:14.986 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/450167167' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-11T20:54:14.986 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: osdmap e87: 8 total, 8 up, 8 in 2024-03-11T20:54:14.986 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/867247583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi129-83754-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:14.987 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: fourxx 2024-03-11T20:54:14.987 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2433590356' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi129-82555-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:14.987 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2387178343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:14.987 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:14.987 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-4-cache", "mode": "writeback"}]: dispatch 2024-03-11T20:54:14.987 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:14.987 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-11T20:54:14.987 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi129-83754-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:14.987 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi129-82555-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:14.987 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:14.987 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:14.987 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: Health check update: 18 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:14.987 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: pool 'PoolQuotaPP_smithi129-82567-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-11T20:54:14.987 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:54:14.988 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-11T20:54:14.988 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:14.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: pgmap v83: 860 pgs: 90 creating+peering, 62 unknown, 708 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:54:14.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:14.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:14.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]': finished 2024-03-11T20:54:14.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-4", "overlaypool": "test-rados-api-smithi129-82818-4-cache"}]': finished 2024-03-11T20:54:14.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi129-83754-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:14.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-11T20:54:14.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-4-cache", "mode": "writeback"}]: dispatch 2024-03-11T20:54:14.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2237768536' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:14.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/450167167' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-11T20:54:14.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: osdmap e87: 8 total, 8 up, 8 in 2024-03-11T20:54:14.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/867247583' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi129-83754-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:14.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: fourxx 2024-03-11T20:54:14.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2433590356' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi129-82555-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:14.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2387178343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:14.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:14.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-4-cache", "mode": "writeback"}]: dispatch 2024-03-11T20:54:14.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:14.990 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-11T20:54:14.990 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi129-83754-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:14.990 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi129-82555-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:14.990 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:14.990 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:14.990 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: Health check update: 18 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:14.990 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: pool 'PoolQuotaPP_smithi129-82567-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-11T20:54:14.990 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:54:14.990 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-11T20:54:14.990 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:15.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-4-cache", "mode": "writeback"}]': finished 2024-03-11T20:54:15.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-83101-1"}]': finished 2024-03-11T20:54:15.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi129-82555-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:15.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:15.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-4"}]: dispatch 2024-03-11T20:54:15.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:15.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:15.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: osdmap e88: 8 total, 8 up, 8 in 2024-03-11T20:54:15.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-11T20:54:15.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:15.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi129-83703-2"}]: dispatch 2024-03-11T20:54:15.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi129-83754-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi129-83754-2"}]': finished 2024-03-11T20:54:15.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi129-82594-16"}]': finished 2024-03-11T20:54:15.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi129-83703-2"}]': finished 2024-03-11T20:54:15.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: osdmap e89: 8 total, 8 up, 8 in 2024-03-11T20:54:15.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:15.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi129-83703-2"}]: dispatch 2024-03-11T20:54:15.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:15.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:15.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:15.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:15.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:15.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:15.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:15.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:15.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:15.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:15.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:15.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-4"}]: dispatch 2024-03-11T20:54:16.104 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-4-cache", "mode": "writeback"}]': finished 2024-03-11T20:54:16.104 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-83101-1"}]': finished 2024-03-11T20:54:16.104 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi129-82555-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:16.104 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:16.104 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-4"}]: dispatch 2024-03-11T20:54:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: osdmap e88: 8 total, 8 up, 8 in 2024-03-11T20:54:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-11T20:54:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi129-83703-2"}]: dispatch 2024-03-11T20:54:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi129-83754-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi129-83754-2"}]': finished 2024-03-11T20:54:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi129-82594-16"}]': finished 2024-03-11T20:54:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi129-83703-2"}]': finished 2024-03-11T20:54:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: osdmap e89: 8 total, 8 up, 8 in 2024-03-11T20:54:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi129-83703-2"}]: dispatch 2024-03-11T20:54:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:16.106 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:16.106 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:16.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-4-cache", "mode": "writeback"}]': finished 2024-03-11T20:54:16.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi129-83101-1"}]': finished 2024-03-11T20:54:16.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi129-82555-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:16.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:16.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-4"}]: dispatch 2024-03-11T20:54:16.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:16.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:16.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: osdmap e88: 8 total, 8 up, 8 in 2024-03-11T20:54:16.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-11T20:54:16.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:16.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi129-83703-2"}]: dispatch 2024-03-11T20:54:16.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi129-83754-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi129-83754-2"}]': finished 2024-03-11T20:54:16.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi129-82594-16"}]': finished 2024-03-11T20:54:16.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi129-83703-2"}]': finished 2024-03-11T20:54:16.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: osdmap e89: 8 total, 8 up, 8 in 2024-03-11T20:54:16.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi129-82594-16"}]: dispatch 2024-03-11T20:54:16.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi129-83703-2"}]: dispatch 2024-03-11T20:54:16.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:16.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:16.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:16.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:16.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:16.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:16.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:16.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:16.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:16.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:16.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:16.108 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-4"}]: dispatch 2024-03-11T20:54:16.108 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:16.108 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:16.109 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:16.109 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:16.109 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:16.109 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:16.109 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:16.109 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:16.109 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-4"}]: dispatch 2024-03-11T20:54:16.206 INFO:tasks.workunit.client.0.smithi129.stdout: ec_io: Running main() from gmock_main.cc 2024-03-11T20:54:16.206 INFO:tasks.workunit.client.0.smithi129.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-03-11T20:54:16.206 INFO:tasks.workunit.client.0.smithi129.stdout: ec_io: [----------] Global test environment set-up. 2024-03-11T20:54:16.206 INFO:tasks.workunit.client.0.smithi129.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-03-11T20:54:16.206 INFO:tasks.workunit.client.0.smithi129.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-03-11T20:54:16.206 INFO:tasks.workunit.client.0.smithi129.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (14987 ms) 2024-03-11T20:54:16.206 INFO:tasks.workunit.client.0.smithi129.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-03-11T20:54:16.206 INFO:tasks.workunit.client.0.smithi129.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (7162 ms) 2024-03-11T20:54:16.207 INFO:tasks.workunit.client.0.smithi129.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (22149 ms total) 2024-03-11T20:54:16.207 INFO:tasks.workunit.client.0.smithi129.stdout: ec_io: 2024-03-11T20:54:16.207 INFO:tasks.workunit.client.0.smithi129.stdout: ec_io: [----------] Global test environment tear-down 2024-03-11T20:54:16.207 INFO:tasks.workunit.client.0.smithi129.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (22149 ms total) 2024-03-11T20:54:16.207 INFO:tasks.workunit.client.0.smithi129.stdout: ec_io: [ PASSED ] 2 tests. 2024-03-11T20:54:16.221 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: Running main() from gmock_main.cc 2024-03-11T20:54:16.221 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-03-11T20:54:16.221 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] Global test environment set-up. 2024-03-11T20:54:16.221 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-03-11T20:54:16.221 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-03-11T20:54:16.221 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (368 ms) 2024-03-11T20:54:16.221 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-03-11T20:54:16.221 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-03-11T20:54:16.221 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-03-11T20:54:16.222 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: no timeout :/ 2024-03-11T20:54:16.222 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: no timeout :/ 2024-03-11T20:54:16.222 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: no timeout :/ 2024-03-11T20:54:16.222 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: no timeout :/ 2024-03-11T20:54:16.222 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: no timeout :/ 2024-03-11T20:54:16.222 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (48 ms) 2024-03-11T20:54:16.222 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-03-11T20:54:16.222 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (9 ms) 2024-03-11T20:54:16.222 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-03-11T20:54:16.222 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.Checksum (3 ms) 2024-03-11T20:54:16.222 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-03-11T20:54:16.222 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (5 ms) 2024-03-11T20:54:16.222 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-03-11T20:54:16.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (6 ms) 2024-03-11T20:54:16.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-03-11T20:54:16.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (4 ms) 2024-03-11T20:54:16.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-03-11T20:54:16.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2 ms) 2024-03-11T20:54:16.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-03-11T20:54:16.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-03-11T20:54:16.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-03-11T20:54:16.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (6 ms) 2024-03-11T20:54:16.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-03-11T20:54:16.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (4 ms) 2024-03-11T20:54:16.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-03-11T20:54:16.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.RmXattr (16 ms) 2024-03-11T20:54:16.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-03-11T20:54:16.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIo.XattrIter (6 ms) 2024-03-11T20:54:16.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] 14 tests from LibRadosIo (482 ms total) 2024-03-11T20:54:16.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: 2024-03-11T20:54:16.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-03-11T20:54:16.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-03-11T20:54:16.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1164 ms) 2024-03-11T20:54:16.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-03-11T20:54:16.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (7 ms) 2024-03-11T20:54:16.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-03-11T20:54:16.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (8 ms) 2024-03-11T20:54:16.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-03-11T20:54:16.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (7 ms) 2024-03-11T20:54:16.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-03-11T20:54:16.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (9 ms) 2024-03-11T20:54:16.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-03-11T20:54:16.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (4 ms) 2024-03-11T20:54:16.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-03-11T20:54:16.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3 ms) 2024-03-11T20:54:16.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-03-11T20:54:16.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (3 ms) 2024-03-11T20:54:16.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-03-11T20:54:16.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (9 ms) 2024-03-11T20:54:16.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-03-11T20:54:16.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4 ms) 2024-03-11T20:54:16.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1219 ms total) 2024-03-11T20:54:16.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: 2024-03-11T20:54:16.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [----------] Global test environment tear-down 2024-03-11T20:54:16.226 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [==========] 24 tests from 2 test suites ran. (22548 ms total) 2024-03-11T20:54:16.226 INFO:tasks.workunit.client.0.smithi129.stdout: api_io: [ PASSED ] 24 tests. 2024-03-11T20:54:16.226 INFO:tasks.workunit.client.0.smithi129.stdout: list: Running main() from gmock_main.cc 2024-03-11T20:54:16.226 INFO:tasks.workunit.client.0.smithi129.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-03-11T20:54:16.226 INFO:tasks.workunit.client.0.smithi129.stdout: list: [----------] Global test environment set-up. 2024-03-11T20:54:16.226 INFO:tasks.workunit.client.0.smithi129.stdout: list: [----------] 3 tests from NeoradosList 2024-03-11T20:54:16.226 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-03-11T20:54:16.226 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ OK ] NeoradosList.ListObjects (10976 ms) 2024-03-11T20:54:16.226 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-03-11T20:54:16.227 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ OK ] NeoradosList.ListObjectsNS (7530 ms) 2024-03-11T20:54:16.227 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-03-11T20:54:16.227 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ OK ] NeoradosList.ListObjectsMany (3666 ms) 2024-03-11T20:54:16.227 INFO:tasks.workunit.client.0.smithi129.stdout: list: [----------] 3 tests from NeoradosList (22172 ms total) 2024-03-11T20:54:16.227 INFO:tasks.workunit.client.0.smithi129.stdout: list: 2024-03-11T20:54:16.227 INFO:tasks.workunit.client.0.smithi129.stdout: list: [----------] Global test environment tear-down 2024-03-11T20:54:16.227 INFO:tasks.workunit.client.0.smithi129.stdout: list: [==========] 3 tests from 1 test suite ran. (22172 ms total) 2024-03-11T20:54:16.227 INFO:tasks.workunit.client.0.smithi129.stdout: list: [ PASSED ] 3 tests. 2024-03-11T20:54:16.478 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:54:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:54:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:54:16.708 INFO:tasks.workunit.client.0.smithi129.stdout:le-LibRadosIoECPP_smithi129-82608-22"}]': finished 2024-03-11T20:54:16.708 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:12.573734+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi129-82918-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi129-82918-7"}]': finished 2024-03-11T20:54:16.708 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:12.603896+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:16.708 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:12.609926+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi129-83101-1"}]: dispatch 2024-03-11T20:54:16.708 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:13.093047+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-4", "overlaypool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:16.709 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:13.094478+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi129-83754-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:16.709 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: got: 2024-03-11T20:54:13.340617+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-11T20:54:16.709 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (11705 ms) 2024-03-11T20:54:16.709 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (22861 ms total) 2024-03-11T20:54:16.709 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: 2024-03-11T20:54:16.709 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [----------] Global test environment tear-down 2024-03-11T20:54:16.709 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (22861 ms total) 2024-03-11T20:54:16.709 INFO:tasks.workunit.client.0.smithi129.stdout: api_cmd: [ PASSED ] 4 tests. 2024-03-11T20:54:17.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: Running main() from gmock_main.cc 2024-03-11T20:54:17.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-03-11T20:54:17.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] Global test environment set-up. 2024-03-11T20:54:17.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-03-11T20:54:17.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-03-11T20:54:17.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStat.Stat (371 ms) 2024-03-11T20:54:17.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-03-11T20:54:17.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (9 ms) 2024-03-11T20:54:17.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-03-11T20:54:17.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStat.StatNS (17 ms) 2024-03-11T20:54:17.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-03-11T20:54:17.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-03-11T20:54:17.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-03-11T20:54:17.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (4 ms) 2024-03-11T20:54:17.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] 5 tests from LibRadosStat (401 ms total) 2024-03-11T20:54:17.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: 2024-03-11T20:54:17.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-03-11T20:54:17.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-03-11T20:54:17.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1134 ms) 2024-03-11T20:54:17.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-03-11T20:54:17.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (12 ms) 2024-03-11T20:54:17.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-03-11T20:54:17.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-03-11T20:54:17.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-03-11T20:54:17.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-03-11T20:54:17.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1148 ms total) 2024-03-11T20:54:17.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: 2024-03-11T20:54:17.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [----------] Global test environment tear-down 2024-03-11T20:54:17.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (23484 ms total) 2024-03-11T20:54:17.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat: [ PASSED ] 9 tests. 2024-03-11T20:54:17.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-03-11T20:54:17.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-03-11T20:54:17.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-03-11T20:54:17.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-03-11T20:54:17.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: seed 83015 2024-03-11T20:54:17.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-03-11T20:54:17.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (390 ms) 2024-03-11T20:54:17.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-03-11T20:54:17.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (9 ms) 2024-03-11T20:54:17.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-03-11T20:54:17.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2024-03-11T20:54:17.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-03-11T20:54:17.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (5 ms) 2024-03-11T20:54:17.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-03-11T20:54:17.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (13 ms) 2024-03-11T20:54:17.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (418 ms total) 2024-03-11T20:54:17.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: 2024-03-11T20:54:17.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-03-11T20:54:17.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-03-11T20:54:17.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1134 ms) 2024-03-11T20:54:17.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-03-11T20:54:17.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1 ms) 2024-03-11T20:54:17.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-03-11T20:54:17.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (4 ms) 2024-03-11T20:54:17.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-03-11T20:54:17.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (10 ms) 2024-03-11T20:54:17.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1149 ms total) 2024-03-11T20:54:17.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: 2024-03-11T20:54:17.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-03-11T20:54:17.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (23469 ms total) 2024-03-11T20:54:17.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-03-11T20:54:17.296 INFO:tasks.workunit.client.0.smithi129.stdout: LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-03-11T20:54:17.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (5 ms) 2024-03-11T20:54:17.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-03-11T20:54:17.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (3 ms) 2024-03-11T20:54:17.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-03-11T20:54:17.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (3 ms) 2024-03-11T20:54:17.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-03-11T20:54:17.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (4 ms) 2024-03-11T20:54:17.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-03-11T20:54:17.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (3 ms) 2024-03-11T20:54:17.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-03-11T20:54:17.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (2 ms) 2024-03-11T20:54:17.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-03-11T20:54:17.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (4 ms) 2024-03-11T20:54:17.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-03-11T20:54:17.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (8 ms) 2024-03-11T20:54:17.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-03-11T20:54:17.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (3 ms) 2024-03-11T20:54:17.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-03-11T20:54:17.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (7 ms) 2024-03-11T20:54:17.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-03-11T20:54:17.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (4 ms) 2024-03-11T20:54:17.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-03-11T20:54:17.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (6 ms) 2024-03-11T20:54:17.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1221 ms total) 2024-03-11T20:54:17.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: 2024-03-11T20:54:17.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [----------] Global test environment tear-down 2024-03-11T20:54:17.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (23618 ms total) 2024-03-11T20:54:17.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-03-11T20:54:17.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: pgmap v86: 780 pgs: 28 creating+peering, 236 unknown, 516 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:54:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi129-82594-16"}]': finished 2024-03-11T20:54:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi129-83703-2"}]': finished 2024-03-11T20:54:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]': finished 2024-03-11T20:54:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]': finished 2024-03-11T20:54:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]': finished 2024-03-11T20:54:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-4"}]': finished 2024-03-11T20:54:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi129-83122-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:17.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:17.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:17.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:17.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: osdmap e90: 8 total, 8 up, 8 in 2024-03-11T20:54:17.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi129-83122-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:17.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:17.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:17.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: pgmap v86: 780 pgs: 28 creating+peering, 236 unknown, 516 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:54:17.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:17.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi129-82594-16"}]': finished 2024-03-11T20:54:17.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi129-83703-2"}]': finished 2024-03-11T20:54:17.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:17.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]': finished 2024-03-11T20:54:17.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]': finished 2024-03-11T20:54:17.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]': finished 2024-03-11T20:54:17.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:17.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-4"}]': finished 2024-03-11T20:54:17.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi129-83122-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:17.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:17.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:17.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:17.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:17.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: osdmap e90: 8 total, 8 up, 8 in 2024-03-11T20:54:17.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi129-83122-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:17.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:17.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:17.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:17.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:17.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:17.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:17.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3986836054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:17.482 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:17.483 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-83101-6", "pg_num": 4}]: dispatch 2024-03-11T20:54:17.483 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:17.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:17.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:17.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:17.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:17.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3986836054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:17.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:17.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-83101-6", "pg_num": 4}]: dispatch 2024-03-11T20:54:17.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: pgmap v86: 780 pgs: 28 creating+peering, 236 unknown, 516 active+clean; 145 MiB data, 734 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:54:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3342771332' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi129-82594-16"}]': finished 2024-03-11T20:54:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2875186884' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi129-83703-2"}]': finished 2024-03-11T20:54:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi129-83015-7"}]': finished 2024-03-11T20:54:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi129-82918-7"}]': finished 2024-03-11T20:54:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi129-82608-22"}]': finished 2024-03-11T20:54:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:17.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-4"}]': finished 2024-03-11T20:54:17.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi129-83122-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:17.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/569934597' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:17.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2528554020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:17.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/966800767' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:17.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3221317149' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:17.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: osdmap e90: 8 total, 8 up, 8 in 2024-03-11T20:54:17.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi129-83122-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:17.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:17.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:17.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]: dispatch 2024-03-11T20:54:17.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]: dispatch 2024-03-11T20:54:17.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]: dispatch 2024-03-11T20:54:17.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]: dispatch 2024-03-11T20:54:17.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3986836054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:17.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:17.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-83101-6", "pg_num": 4}]: dispatch 2024-03-11T20:54:17.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:18.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-03-11T20:54:18.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-03-11T20:54:18.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-03-11T20:54:18.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-03-11T20:54:18.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: seed 82679 2024-03-11T20:54:18.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-03-11T20:54:18.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (373 ms) 2024-03-11T20:54:18.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-03-11T20:54:18.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (6 ms) 2024-03-11T20:54:18.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-03-11T20:54:18.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1010 ms) 2024-03-11T20:54:18.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-03-11T20:54:18.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1041 ms) 2024-03-11T20:54:18.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-03-11T20:54:18.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (48 ms) 2024-03-11T20:54:18.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-03-11T20:54:18.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (28 ms) 2024-03-11T20:54:18.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-03-11T20:54:18.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (7 ms) 2024-03-11T20:54:18.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-03-11T20:54:18.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-03-11T20:54:18.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2516 ms total) 2024-03-11T20:54:18.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: 2024-03-11T20:54:18.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-03-11T20:54:18.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-03-11T20:54:18.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1132 ms) 2024-03-11T20:54:18.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-03-11T20:54:18.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (11 ms) 2024-03-11T20:54:18.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-03-11T20:54:18.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1004 ms) 2024-03-11T20:54:18.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-03-11T20:54:18.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1005 ms) 2024-03-11T20:54:18.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-03-11T20:54:18.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (4 ms) 2024-03-11T20:54:18.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-03-11T20:54:18.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (4 ms) 2024-03-11T20:54:18.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-03-11T20:54:18.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (5 ms) 2024-03-11T20:54:18.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-03-11T20:54:18.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (2 ms) 2024-03-11T20:54:18.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3167 ms total) 2024-03-11T20:54:18.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: 2024-03-11T20:54:18.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-03-11T20:54:18.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (24586 ms total) 2024-03-11T20:54:18.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-03-11T20:54:18.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: Running main() from gmock_main.cc 2024-03-11T20:54:18.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-03-11T20:54:18.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] Global test environment set-up. 2024-03-11T20:54:18.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-03-11T20:54:18.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-03-11T20:54:18.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (356 ms) 2024-03-11T20:54:18.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-03-11T20:54:18.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.LockShared (5 ms) 2024-03-11T20:54:18.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-03-11T20:54:18.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1005 ms) 2024-03-11T20:54:18.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-03-11T20:54:18.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1155 ms) 2024-03-11T20:54:18.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-03-11T20:54:18.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (4 ms) 2024-03-11T20:54:18.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-03-11T20:54:18.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.Unlock (3 ms) 2024-03-11T20:54:18.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-03-11T20:54:18.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (6 ms) 2024-03-11T20:54:18.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-03-11T20:54:18.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (3 ms) 2024-03-11T20:54:18.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] 8 tests from LibRadosLock (2537 ms total) 2024-03-11T20:54:18.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: 2024-03-11T20:54:18.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-03-11T20:54:18.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-03-11T20:54:18.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1127 ms) 2024-03-11T20:54:18.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-03-11T20:54:18.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (14 ms) 2024-03-11T20:54:18.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-03-11T20:54:18.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1006 ms) 2024-03-11T20:54:18.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-03-11T20:54:18.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1004 ms) 2024-03-11T20:54:18.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-03-11T20:54:18.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (4 ms) 2024-03-11T20:54:18.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-03-11T20:54:18.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (4 ms) 2024-03-11T20:54:18.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-03-11T20:54:18.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (5 ms) 2024-03-11T20:54:18.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-03-11T20:54:18.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (3 ms) 2024-03-11T20:54:18.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3167 ms total) 2024-03-11T20:54:18.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: 2024-03-11T20:54:18.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [----------] Global test environment tear-down 2024-03-11T20:54:18.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (24590 ms total) 2024-03-11T20:54:18.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_lock: [ PASSED ] 16 tests. 2024-03-11T20:54:18.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-83101-6", "pg_num": 4}]: dispatch 2024-03-11T20:54:18.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:54:18.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10"}]': finished 2024-03-11T20:54:18.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi129-82672-10"}]': finished 2024-03-11T20:54:18.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]': finished 2024-03-11T20:54:18.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]': finished 2024-03-11T20:54:18.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]': finished 2024-03-11T20:54:18.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]': finished 2024-03-11T20:54:18.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:18.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-83101-6", "pg_num": 4}]': finished 2024-03-11T20:54:18.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:18.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: osdmap e91: 8 total, 8 up, 8 in 2024-03-11T20:54:18.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2502678732' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:18.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:18.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:18.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:18.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3592597164' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi129-82555-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:18.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:54:18.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:18.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:18.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:18.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-83101-6", "pg_num": 4}]: dispatch 2024-03-11T20:54:18.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:54:18.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10"}]': finished 2024-03-11T20:54:18.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi129-82672-10"}]': finished 2024-03-11T20:54:18.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]': finished 2024-03-11T20:54:18.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]': finished 2024-03-11T20:54:18.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]': finished 2024-03-11T20:54:18.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]': finished 2024-03-11T20:54:18.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:18.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-83101-6", "pg_num": 4}]': finished 2024-03-11T20:54:18.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:18.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: osdmap e91: 8 total, 8 up, 8 in 2024-03-11T20:54:18.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2502678732' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:18.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:18.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:18.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:18.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3592597164' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi129-82555-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:18.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:54:18.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:18.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:18.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:18.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-83101-6", "pg_num": 4}]: dispatch 2024-03-11T20:54:18.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:54:18.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi129-82679-10"}]': finished 2024-03-11T20:54:18.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi129-82672-10"}]': finished 2024-03-11T20:54:18.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-4", "tierpool": "test-rados-api-smithi129-82818-4-cache"}]': finished 2024-03-11T20:54:18.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi129-83015-7"}]': finished 2024-03-11T20:54:18.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi129-82608-22"}]': finished 2024-03-11T20:54:18.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi129-82918-7"}]': finished 2024-03-11T20:54:18.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:18.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-83101-6", "pg_num": 4}]': finished 2024-03-11T20:54:18.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:18.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: osdmap e91: 8 total, 8 up, 8 in 2024-03-11T20:54:18.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2502678732' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:18.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi129-82679-10"}]: dispatch 2024-03-11T20:54:18.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi129-82672-10"}]: dispatch 2024-03-11T20:54:18.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:18.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3592597164' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi129-82555-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:18.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:54:18.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:18.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:18.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:19.334 INFO:tasks.workunit.client.0.smithi129.stdout: pool: Running main() from gmock_main.cc 2024-03-11T20:54:19.334 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-03-11T20:54:19.335 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [----------] Global test environment set-up. 2024-03-11T20:54:19.335 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-03-11T20:54:19.335 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-03-11T20:54:19.335 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ OK ] NeoRadosPools.PoolList (9970 ms) 2024-03-11T20:54:19.335 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-03-11T20:54:19.335 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (3721 ms) 2024-03-11T20:54:19.335 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-03-11T20:54:19.335 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2532 ms) 2024-03-11T20:54:19.335 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-03-11T20:54:19.335 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (4881 ms) 2024-03-11T20:54:19.335 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-03-11T20:54:19.335 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2336 ms) 2024-03-11T20:54:19.335 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-03-11T20:54:19.335 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (1790 ms) 2024-03-11T20:54:19.336 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [----------] 6 tests from NeoRadosPools (25230 ms total) 2024-03-11T20:54:19.336 INFO:tasks.workunit.client.0.smithi129.stdout: pool: 2024-03-11T20:54:19.336 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [----------] Global test environment tear-down 2024-03-11T20:54:19.336 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [==========] 6 tests from 1 test suite ran. (25230 ms total) 2024-03-11T20:54:19.336 INFO:tasks.workunit.client.0.smithi129.stdout: pool: [ PASSED ] 6 tests. 2024-03-11T20:54:19.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[30184]: pgmap v89: 748 pgs: 96 creating+peering, 652 active+clean; 145 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.5 KiB/s wr, 3 op/s 2024-03-11T20:54:19.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi129-83122-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]': finished 2024-03-11T20:54:19.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi129-82679-10"}]': finished 2024-03-11T20:54:19.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi129-82672-10"}]': finished 2024-03-11T20:54:19.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:54:19.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3592597164' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi129-82555-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:19.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:19.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[30184]: osdmap e92: 8 total, 8 up, 8 in 2024-03-11T20:54:19.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "overlaypool": "test-rados-api-smithi129-83101-6"}]: dispatch 2024-03-11T20:54:19.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/867247583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:19.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3157202610' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi129-82945-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:19.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "overlaypool": "test-rados-api-smithi129-83101-6"}]: dispatch 2024-03-11T20:54:19.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:19.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi129-82945-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:19.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:19.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[42066]: pgmap v89: 748 pgs: 96 creating+peering, 652 active+clean; 145 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.5 KiB/s wr, 3 op/s 2024-03-11T20:54:19.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi129-83122-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]': finished 2024-03-11T20:54:19.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi129-82679-10"}]': finished 2024-03-11T20:54:19.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi129-82672-10"}]': finished 2024-03-11T20:54:19.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:54:19.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3592597164' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi129-82555-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:19.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:19.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[42066]: osdmap e92: 8 total, 8 up, 8 in 2024-03-11T20:54:19.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "overlaypool": "test-rados-api-smithi129-83101-6"}]: dispatch 2024-03-11T20:54:19.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/867247583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:19.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3157202610' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi129-82945-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:19.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "overlaypool": "test-rados-api-smithi129-83101-6"}]: dispatch 2024-03-11T20:54:19.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:19.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi129-82945-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:19.734 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:19.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:19 smithi157 ceph-mon[33359]: pgmap v89: 748 pgs: 96 creating+peering, 652 active+clean; 145 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.5 KiB/s wr, 3 op/s 2024-03-11T20:54:19.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi129-83122-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]': finished 2024-03-11T20:54:19.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1547548630' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi129-82679-10"}]': finished 2024-03-11T20:54:19.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1550925160' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi129-82672-10"}]': finished 2024-03-11T20:54:19.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:54:19.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3592597164' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi129-82555-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:19.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:19.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:19 smithi157 ceph-mon[33359]: osdmap e92: 8 total, 8 up, 8 in 2024-03-11T20:54:19.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "overlaypool": "test-rados-api-smithi129-83101-6"}]: dispatch 2024-03-11T20:54:19.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/867247583' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:19.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3157202610' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi129-82945-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:19.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "overlaypool": "test-rados-api-smithi129-83101-6"}]: dispatch 2024-03-11T20:54:19.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:19.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi129-82945-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:19.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:20.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: pgmap v92: 688 pgs: 204 unknown, 484 active+clean; 145 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 2 op/s 2024-03-11T20:54:20.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "overlaypool": "test-rados-api-smithi129-83101-6"}]': finished 2024-03-11T20:54:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi129-83754-2"}]': finished 2024-03-11T20:54:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi129-82945-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-83101-6", "mode": "writeback"}]: dispatch 2024-03-11T20:54:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/867247583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: osdmap e93: 8 total, 8 up, 8 in 2024-03-11T20:54:20.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-83101-6", "mode": "writeback"}]: dispatch 2024-03-11T20:54:20.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:20.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2863590076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:20.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:54:20.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-11T20:54:20.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-83101-6", "mode": "writeback"}]': finished 2024-03-11T20:54:20.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi129-83754-2"}]': finished 2024-03-11T20:54:20.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:20.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: osdmap e94: 8 total, 8 up, 8 in 2024-03-11T20:54:20.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2527947756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi129-82555-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:20.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi129-82555-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:20.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/776140368' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.733 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: pgmap v92: 688 pgs: 204 unknown, 484 active+clean; 145 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 2 op/s 2024-03-11T20:54:20.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:20.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "overlaypool": "test-rados-api-smithi129-83101-6"}]': finished 2024-03-11T20:54:20.734 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi129-83754-2"}]': finished 2024-03-11T20:54:20.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi129-82945-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:20.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-83101-6", "mode": "writeback"}]: dispatch 2024-03-11T20:54:20.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/867247583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:20.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: osdmap e93: 8 total, 8 up, 8 in 2024-03-11T20:54:20.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-83101-6", "mode": "writeback"}]: dispatch 2024-03-11T20:54:20.735 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:20.736 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2863590076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.736 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.736 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:20.736 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:54:20.736 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-11T20:54:20.736 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-83101-6", "mode": "writeback"}]': finished 2024-03-11T20:54:20.737 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi129-83754-2"}]': finished 2024-03-11T20:54:20.737 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:20.737 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: osdmap e94: 8 total, 8 up, 8 in 2024-03-11T20:54:20.737 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2527947756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi129-82555-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.737 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:20.737 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi129-82555-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.737 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:20.738 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/776140368' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.738 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: pgmap v92: 688 pgs: 204 unknown, 484 active+clean; 145 MiB data, 764 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 2 op/s 2024-03-11T20:54:20.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:20.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "overlaypool": "test-rados-api-smithi129-83101-6"}]': finished 2024-03-11T20:54:20.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi129-83754-2"}]': finished 2024-03-11T20:54:20.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi129-82945-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:20.740 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-83101-6", "mode": "writeback"}]: dispatch 2024-03-11T20:54:20.740 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/867247583' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:20.740 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: osdmap e93: 8 total, 8 up, 8 in 2024-03-11T20:54:20.740 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-83101-6", "mode": "writeback"}]: dispatch 2024-03-11T20:54:20.740 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi129-83754-2"}]: dispatch 2024-03-11T20:54:20.740 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2863590076' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.741 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.741 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:20.741 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:54:20.741 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-11T20:54:20.741 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-83101-6", "mode": "writeback"}]': finished 2024-03-11T20:54:20.741 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi129-83754-2"}]': finished 2024-03-11T20:54:20.742 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:20.742 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: osdmap e94: 8 total, 8 up, 8 in 2024-03-11T20:54:20.742 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2527947756' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi129-82555-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.742 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:20.742 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi129-82555-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.742 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:20.743 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/776140368' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:20.743 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:21.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/944285506' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi129-83754-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:21.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:21.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi129-83754-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:21.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi129-82555-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:21.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]': finished 2024-03-11T20:54:21.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:21.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi129-83754-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:21.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[42066]: osdmap e95: 8 total, 8 up, 8 in 2024-03-11T20:54:21.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/944285506' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi129-83754-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:21.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:21.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi129-83754-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:21.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:21.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:21.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:21.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/944285506' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi129-83754-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:21.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:21.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi129-83754-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:21.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi129-82555-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:21.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]': finished 2024-03-11T20:54:21.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:21.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi129-83754-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:21.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[30184]: osdmap e95: 8 total, 8 up, 8 in 2024-03-11T20:54:21.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/944285506' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi129-83754-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:21.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:21.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi129-83754-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:21.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:21.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:21.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:21.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/944285506' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi129-83754-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:21.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:21.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi129-83754-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:21.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi129-82555-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:21.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi129-83122-12"}]': finished 2024-03-11T20:54:21.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:21.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi129-83754-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:21.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:21 smithi157 ceph-mon[33359]: osdmap e95: 8 total, 8 up, 8 in 2024-03-11T20:54:21.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/944285506' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi129-83754-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:21.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2701544175' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:21.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi129-83754-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:21.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-83122-12"}]: dispatch 2024-03-11T20:54:21.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:21.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:22.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-03-11T20:54:22.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-03-11T20:54:22.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-03-11T20:54:22.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-03-11T20:54:22.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-03-11T20:54:22.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify_test_cb 2024-03-11T20:54:22.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (397 ms) 2024-03-11T20:54:22.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-03-11T20:54:22.389 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-11T20:54:22.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94801976431216 err -107 2024-03-11T20:54:22.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1015 ms) 2024-03-11T20:54:22.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-03-11T20:54:22.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_errcb cookie waiting up to 94801976442960300 err -107 for disconnect notification ... 2024-03-11T20:54:22.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: 2024-03-11T20:54:22.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (6 ms) 2024-03-11T20:54:22.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-03-11T20:54:22.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 24713 notify_id 339302416385 cookie 94801976446240 2024-03-11T20:54:22.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (6 ms) 2024-03-11T20:54:22.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-03-11T20:54:22.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 24713 notify_id 339302416386 cookie 94801976484896 2024-03-11T20:54:22.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (6 ms) 2024-03-11T20:54:22.390 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-03-11T20:54:22.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 24713 notify_id 339302416386 cookie 94801976484896 2024-03-11T20:54:22.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (7 ms) 2024-03-11T20:54:22.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-03-11T20:54:22.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 24713 notify_id 339302416384 cookie 94801976484896 2024-03-11T20:54:22.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 24713 notify_id 339302416384 cookie 94801976495408 2024-03-11T20:54:22.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (6 ms) 2024-03-11T20:54:22.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-03-11T20:54:22.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 24713 notify_id 339302416387 cookie 94801976484896 2024-03-11T20:54:22.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 24713 notify_id 343597383684 cookie 94801976484896 2024-03-11T20:54:22.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3006 ms) 2024-03-11T20:54:22.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-03-11T20:54:22.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-03-11T20:54:22.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94801976484896 err -107 2024-03-11T20:54:22.391 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_cb from 24713 notify_id 365072220161 cookie 94801976484896 2024-03-11T20:54:22.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5007 ms) 2024-03-11T20:54:22.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-03-11T20:54:22.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-03-11T20:54:22.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94801976495408 err -107 2024-03-11T20:54:22.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1004 ms) 2024-03-11T20:54:22.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (10460 ms total) 2024-03-11T20:54:22.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: 2024-03-11T20:54:22.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-03-11T20:54:22.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-03-11T20:54:22.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: watch_notify_test_cb 2024-03-11T20:54:22.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1143 ms) 2024-03-11T20:54:22.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1143 ms total) 2024-03-11T20:54:22.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: 2024-03-11T20:54:22.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-03-11T20:54:22.392 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (28543 ms total) 2024-03-11T20:54:22.393 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-03-11T20:54:23.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:22 smithi129 ceph-mon[30184]: pgmap v95: 648 pgs: 260 unknown, 388 active+clean; 145 MiB data, 764 MiB used, 714 GiB / 715 GiB avail 2024-03-11T20:54:23.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:23.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:22 smithi129 ceph-mon[42066]: pgmap v95: 648 pgs: 260 unknown, 388 active+clean; 145 MiB data, 764 MiB used, 714 GiB / 715 GiB avail 2024-03-11T20:54:23.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:23.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:22 smithi157 ceph-mon[33359]: pgmap v95: 648 pgs: 260 unknown, 388 active+clean; 145 MiB data, 764 MiB used, 714 GiB / 715 GiB avail 2024-03-11T20:54:23.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:23.485 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-03-11T20:54:23.485 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-03-11T20:54:23.485 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-03-11T20:54:23.485 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-03-11T20:54:23.486 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-03-11T20:54:23.486 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify 2024-03-11T20:54:23.486 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (2873 ms) 2024-03-11T20:54:23.486 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-03-11T20:54:23.486 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (1370 ms) 2024-03-11T20:54:23.486 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (4243 ms total) 2024-03-11T20:54:23.486 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: 2024-03-11T20:54:23.486 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-03-11T20:54:23.486 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-03-11T20:54:23.486 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify 2024-03-11T20:54:23.486 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (138 ms) 2024-03-11T20:54:23.486 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-03-11T20:54:23.486 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify 2024-03-11T20:54:23.487 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (4090 ms) 2024-03-11T20:54:23.487 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-03-11T20:54:23.487 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (10 ms) 2024-03-11T20:54:23.487 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-03-11T20:54:23.487 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (4 ms) 2024-03-11T20:54:23.487 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-03-11T20:54:23.487 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94296022072944 notify_id 403726925826 notifier_gid 25262 2024-03-11T20:54:23.487 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (5 ms) 2024-03-11T20:54:23.487 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-03-11T20:54:23.487 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94296022072944 notify_id 403726925826 notifier_gid 25262 2024-03-11T20:54:23.487 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (4 ms) 2024-03-11T20:54:23.487 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-03-11T20:54:23.487 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94296022078464 notify_id 403726925827 notifier_gid 25262 2024-03-11T20:54:23.487 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (5 ms) 2024-03-11T20:54:23.488 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-03-11T20:54:23.488 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94296022078464 notify_id 403726925825 notifier_gid 25262 2024-03-11T20:54:23.488 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (5 ms) 2024-03-11T20:54:23.488 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-03-11T20:54:23.488 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94296022078464 notify_id 403726925827 notifier_gid 25262 2024-03-11T20:54:23.488 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (4 ms) 2024-03-11T20:54:23.488 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-03-11T20:54:23.488 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94296022091792 notify_id 403726925827 notifier_gid 25262 2024-03-11T20:54:23.488 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (4 ms) 2024-03-11T20:54:23.488 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-03-11T20:54:23.488 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: trying... 2024-03-11T20:54:23.488 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94296022091792 notify_id 403726925828 notifier_gid 25262 2024-03-11T20:54:23.488 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: timed out 2024-03-11T20:54:23.488 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: flushing 2024-03-11T20:54:23.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:54:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:54:24.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-83122-12"}]': finished 2024-03-11T20:54:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: osdmap e96: 8 total, 8 up, 8 in 2024-03-11T20:54:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3723235448' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3350184795' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi129-82567-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi129-83754-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi129-83754-3"}]': finished 2024-03-11T20:54:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3350184795' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi129-82567-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: osdmap e97: 8 total, 8 up, 8 in 2024-03-11T20:54:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:24.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3582546615' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi129-82555-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3692778186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi129-82913-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi129-82555-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi129-82913-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-83122-12"}]': finished 2024-03-11T20:54:24.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:24.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: osdmap e96: 8 total, 8 up, 8 in 2024-03-11T20:54:24.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3723235448' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:24.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3350184795' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi129-82567-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi129-83754-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi129-83754-3"}]': finished 2024-03-11T20:54:24.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3350184795' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi129-82567-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:24.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:24.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: osdmap e97: 8 total, 8 up, 8 in 2024-03-11T20:54:24.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:24.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3582546615' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi129-82555-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3692778186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi129-82913-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi129-82555-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi129-82913-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi129-83122-12"}]': finished 2024-03-11T20:54:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: osdmap e96: 8 total, 8 up, 8 in 2024-03-11T20:54:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3723235448' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3350184795' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi129-82567-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi129-83754-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi129-83754-3"}]': finished 2024-03-11T20:54:24.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3350184795' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi129-82567-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:24.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:24.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: osdmap e97: 8 total, 8 up, 8 in 2024-03-11T20:54:24.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:24.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3582546615' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi129-82555-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3692778186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi129-82913-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi129-82555-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:24.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi129-82913-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:25.394 INFO:tasks.workunit.client.0.smithi129.stdout: api_dry: 1 daemon active (1 hosts) 2024-03-11T20:54:25.394 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-11T20:54:25.394 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-03-11T20:54:25.394 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: data: 2024-03-11T20:54:25.395 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pools: 38 pools, 1156 pgs 2024-03-11T20:54:25.395 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: objects: 198 objects, 582 KiB 2024-03-11T20:54:25.395 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: usage: 232 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:54:25.395 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: pgs: 13.841% pgs unknown 2024-03-11T20:54:25.395 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 73.616% pgs not active 2024-03-11T20:54:25.395 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 851 creating+peering 2024-03-11T20:54:25.395 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 160 unknown 2024-03-11T20:54:25.395 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 145 active+clean 2024-03-11T20:54:25.395 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-03-11T20:54:25.395 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: io: 2024-03-11T20:54:25.395 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-11T20:54:25.396 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-03-11T20:54:25.396 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-03-11T20:54:25.396 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ OK ] LibRadosService.StatusFormat (6373 ms) 2024-03-11T20:54:25.396 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ RUN ] LibRadosService.Status 2024-03-11T20:54:25.396 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ OK ] LibRadosService.Status (20040 ms) 2024-03-11T20:54:25.396 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [----------] 4 tests from LibRadosService (31499 ms total) 2024-03-11T20:54:25.396 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: 2024-03-11T20:54:25.396 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [----------] Global test environment tear-down 2024-03-11T20:54:25.396 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31499 ms total) 2024-03-11T20:54:25.396 INFO:tasks.workunit.client.0.smithi129.stdout: api_service: [ PASSED ] 4 tests. 2024-03-11T20:54:25.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[42066]: pgmap v98: 616 pgs: 64 creating+peering, 128 unknown, 424 active+clean; 145 MiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-03-11T20:54:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[42066]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi129-82555-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi129-82913-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[42066]: osdmap e98: 8 total, 8 up, 8 in 2024-03-11T20:54:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/833730477' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:25.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[30184]: pgmap v98: 616 pgs: 64 creating+peering, 128 unknown, 424 active+clean; 145 MiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-03-11T20:54:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[30184]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi129-82555-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi129-82913-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[30184]: osdmap e98: 8 total, 8 up, 8 in 2024-03-11T20:54:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/833730477' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:25 smithi157 ceph-mon[33359]: pgmap v98: 616 pgs: 64 creating+peering, 128 unknown, 424 active+clean; 145 MiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-03-11T20:54:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:25 smithi157 ceph-mon[33359]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi129-82555-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi129-82913-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:25 smithi157 ceph-mon[33359]: osdmap e98: 8 total, 8 up, 8 in 2024-03-11T20:54:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/833730477' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:26.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-03-11T20:54:26.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [----------] Global test environment set-up. 2024-03-11T20:54:26.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-03-11T20:54:26.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-03-11T20:54:26.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5072 ms) 2024-03-11T20:54:26.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-03-11T20:54:26.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (1565 ms) 2024-03-11T20:54:26.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-03-11T20:54:26.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20709 ms) 2024-03-11T20:54:26.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-03-11T20:54:26.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: attempt 0 of 20 2024-03-11T20:54:26.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5045 ms) 2024-03-11T20:54:26.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (32391 ms total) 2024-03-11T20:54:26.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: 2024-03-11T20:54:26.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [----------] Global test environment tear-down 2024-03-11T20:54:26.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (32391 ms total) 2024-03-11T20:54:26.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-03-11T20:54:26.478 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:54:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:54:26.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[42066]: pgmap v101: 624 pgs: 32 creating+peering, 232 unknown, 360 active+clean; 145 MiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-11T20:54:26.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:26.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:26.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/833730477' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:26.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:54:26.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[42066]: osdmap e99: 8 total, 8 up, 8 in 2024-03-11T20:54:26.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-7"}]: dispatch 2024-03-11T20:54:26.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-7"}]: dispatch 2024-03-11T20:54:26.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2639742453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi129-82567-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:26.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi129-82567-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:26.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[30184]: pgmap v101: 624 pgs: 32 creating+peering, 232 unknown, 360 active+clean; 145 MiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-11T20:54:26.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:26.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:26.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/833730477' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:26.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:54:26.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[30184]: osdmap e99: 8 total, 8 up, 8 in 2024-03-11T20:54:26.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-7"}]: dispatch 2024-03-11T20:54:26.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-7"}]: dispatch 2024-03-11T20:54:26.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2639742453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi129-82567-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:26.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:26 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi129-82567-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:26.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:26 smithi157 ceph-mon[33359]: pgmap v101: 624 pgs: 32 creating+peering, 232 unknown, 360 active+clean; 145 MiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-11T20:54:26.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:26 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:26.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:26 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:26.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:26 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/833730477' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:26.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:26 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:54:26.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:26 smithi157 ceph-mon[33359]: osdmap e99: 8 total, 8 up, 8 in 2024-03-11T20:54:26.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:26 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-7"}]: dispatch 2024-03-11T20:54:26.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:26 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-7"}]: dispatch 2024-03-11T20:54:26.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:26 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2639742453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi129-82567-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:26.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:26 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi129-82567-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:27.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2944262715' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-11T20:54:27.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-11T20:54:27.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:27.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-7"}]': finished 2024-03-11T20:54:27.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi129-82567-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:27.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[30184]: osdmap e100: 8 total, 8 up, 8 in 2024-03-11T20:54:27.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3745363086' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi129-82555-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:27.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi129-82555-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:27.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/944285506' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:27.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:27.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:27.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2944262715' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-11T20:54:27.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-11T20:54:27.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:27.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-7"}]': finished 2024-03-11T20:54:27.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi129-82567-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:27.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[42066]: osdmap e100: 8 total, 8 up, 8 in 2024-03-11T20:54:27.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3745363086' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi129-82555-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:27.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi129-82555-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:27.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/944285506' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:27.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:27.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:27.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2944262715' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-11T20:54:27.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:27 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-11T20:54:27.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:27.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:27 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-7"}]': finished 2024-03-11T20:54:27.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:27 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi129-82567-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:27.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:27 smithi157 ceph-mon[33359]: osdmap e100: 8 total, 8 up, 8 in 2024-03-11T20:54:27.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3745363086' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi129-82555-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:27.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:27 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi129-82555-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:27.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/944285506' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:27.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:27 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:27.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:28.763 INFO:tasks.workunit.client.0.smithi129.stdout: ec_list: Running main() from gmock_main.cc 2024-03-11T20:54:28.763 INFO:tasks.workunit.client.0.smithi129.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-03-11T20:54:28.763 INFO:tasks.workunit.client.0.smithi129.stdout: ec_list: [----------] Global test environment set-up. 2024-03-11T20:54:28.764 INFO:tasks.workunit.client.0.smithi129.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-03-11T20:54:28.764 INFO:tasks.workunit.client.0.smithi129.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-03-11T20:54:28.764 INFO:tasks.workunit.client.0.smithi129.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (17008 ms) 2024-03-11T20:54:28.764 INFO:tasks.workunit.client.0.smithi129.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-03-11T20:54:28.764 INFO:tasks.workunit.client.0.smithi129.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (9250 ms) 2024-03-11T20:54:28.764 INFO:tasks.workunit.client.0.smithi129.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-03-11T20:54:28.764 INFO:tasks.workunit.client.0.smithi129.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (8421 ms) 2024-03-11T20:54:28.764 INFO:tasks.workunit.client.0.smithi129.stdout: ec_list: [----------] 3 tests from NeoradosECList (34679 ms total) 2024-03-11T20:54:28.764 INFO:tasks.workunit.client.0.smithi129.stdout: ec_list: 2024-03-11T20:54:28.764 INFO:tasks.workunit.client.0.smithi129.stdout: ec_list: [----------] Global test environment tear-down 2024-03-11T20:54:28.764 INFO:tasks.workunit.client.0.smithi129.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (34679 ms total) 2024-03-11T20:54:28.764 INFO:tasks.workunit.client.0.smithi129.stdout: ec_list: [ PASSED ] 3 tests. 2024-03-11T20:54:28.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[30184]: pgmap v104: 648 pgs: 96 creating+peering, 64 unknown, 488 active+clean; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-11T20:54:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi129-82555-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi129-83754-3"}]': finished 2024-03-11T20:54:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:54:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[30184]: osdmap e101: 8 total, 8 up, 8 in 2024-03-11T20:54:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7"}]: dispatch 2024-03-11T20:54:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/944285506' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1221964399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7"}]: dispatch 2024-03-11T20:54:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:28.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:28.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1233670755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[42066]: pgmap v104: 648 pgs: 96 creating+peering, 64 unknown, 488 active+clean; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-11T20:54:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi129-82555-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi129-83754-3"}]': finished 2024-03-11T20:54:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:54:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[42066]: osdmap e101: 8 total, 8 up, 8 in 2024-03-11T20:54:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7"}]: dispatch 2024-03-11T20:54:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/944285506' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1221964399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:28.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7"}]: dispatch 2024-03-11T20:54:28.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:28.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:28.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1233670755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:28 smithi157 ceph-mon[33359]: pgmap v104: 648 pgs: 96 creating+peering, 64 unknown, 488 active+clean; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-11T20:54:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi129-82555-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi129-83754-3"}]': finished 2024-03-11T20:54:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:54:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:28 smithi157 ceph-mon[33359]: osdmap e101: 8 total, 8 up, 8 in 2024-03-11T20:54:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7"}]: dispatch 2024-03-11T20:54:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/944285506' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:28.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1221964399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:28.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7"}]: dispatch 2024-03-11T20:54:28.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi129-83754-3"}]: dispatch 2024-03-11T20:54:28.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:28.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1233670755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:29.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:29.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7"}]': finished 2024-03-11T20:54:29.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi129-83754-3"}]': finished 2024-03-11T20:54:29.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:29.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1233670755' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:29.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:29 smithi129 ceph-mon[30184]: osdmap e102: 8 total, 8 up, 8 in 2024-03-11T20:54:29.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2878077953' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi129-82567-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:29.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi129-82567-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:29.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:29.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7"}]': finished 2024-03-11T20:54:29.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi129-83754-3"}]': finished 2024-03-11T20:54:29.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:29.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1233670755' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:29.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:29 smithi129 ceph-mon[42066]: osdmap e102: 8 total, 8 up, 8 in 2024-03-11T20:54:29.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2878077953' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi129-82567-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:29.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi129-82567-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:29.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:29.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-7"}]': finished 2024-03-11T20:54:29.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi129-83754-3"}]': finished 2024-03-11T20:54:29.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-83230-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:29.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1233670755' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:29.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:29 smithi157 ceph-mon[33359]: osdmap e102: 8 total, 8 up, 8 in 2024-03-11T20:54:29.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2878077953' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi129-82567-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:29.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi129-82567-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:30.646 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-03-11T20:54:30.646 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-03-11T20:54:30.646 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-03-11T20:54:30.646 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-03-11T20:54:30.646 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-03-11T20:54:30.646 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-03-11T20:54:30.646 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-03-11T20:54:30.646 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: 2024-03-11T20:54:30.646 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-03-11T20:54:30.646 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-03-11T20:54:30.647 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (10195 ms) 2024-03-11T20:54:30.647 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-03-11T20:54:30.647 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (6231 ms) 2024-03-11T20:54:30.647 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-03-11T20:54:30.647 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (5892 ms) 2024-03-11T20:54:30.647 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-03-11T20:54:30.647 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3125 ms) 2024-03-11T20:54:30.647 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-03-11T20:54:30.647 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (4301 ms) 2024-03-11T20:54:30.647 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-03-11T20:54:30.647 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3114 ms) 2024-03-11T20:54:30.647 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-03-11T20:54:30.647 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (3885 ms) 2024-03-11T20:54:30.647 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (36743 ms total) 2024-03-11T20:54:30.648 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: 2024-03-11T20:54:30.648 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-03-11T20:54:30.648 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (36743 ms total) 2024-03-11T20:54:30.648 INFO:tasks.workunit.client.0.smithi129.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-03-11T20:54:30.648 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: Running main() from gmock_main.cc 2024-03-11T20:54:30.648 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-03-11T20:54:30.648 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [----------] Global test environment set-up. 2024-03-11T20:54:30.648 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-03-11T20:54:30.648 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-03-11T20:54:30.649 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (10874 ms) 2024-03-11T20:54:30.649 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-03-11T20:54:30.649 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (8609 ms) 2024-03-11T20:54:30.649 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-03-11T20:54:30.649 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3935 ms) 2024-03-11T20:54:30.649 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-03-11T20:54:30.649 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (2793 ms) 2024-03-11T20:54:30.649 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-03-11T20:54:30.649 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3317 ms) 2024-03-11T20:54:30.649 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-03-11T20:54:30.649 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3104 ms) 2024-03-11T20:54:30.649 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-03-11T20:54:30.649 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (3878 ms) 2024-03-11T20:54:30.649 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (36511 ms total) 2024-03-11T20:54:30.649 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: 2024-03-11T20:54:30.650 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [----------] Global test environment tear-down 2024-03-11T20:54:30.650 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (36511 ms total) 2024-03-11T20:54:30.650 INFO:tasks.workunit.client.0.smithi129.stdout: write_operations: [ PASSED ] 7 tests. 2024-03-11T20:54:31.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[42066]: pgmap v107: 680 pgs: 192 unknown, 488 active+clean; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T20:54:31.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[42066]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:31.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi129-82567-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:31.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[42066]: osdmap e103: 8 total, 8 up, 8 in 2024-03-11T20:54:31.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3017803857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi129-82555-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:31.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi129-82555-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:31.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:31.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi129-82555-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:31.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[42066]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T20:54:31.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[30184]: pgmap v107: 680 pgs: 192 unknown, 488 active+clean; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T20:54:31.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[30184]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:31.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi129-82567-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:31.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[30184]: osdmap e103: 8 total, 8 up, 8 in 2024-03-11T20:54:31.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3017803857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi129-82555-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:31.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi129-82555-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:31.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:31.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi129-82555-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:31.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:31 smithi129 ceph-mon[30184]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T20:54:31.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:31 smithi157 ceph-mon[33359]: pgmap v107: 680 pgs: 192 unknown, 488 active+clean; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T20:54:31.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:31 smithi157 ceph-mon[33359]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:31.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi129-82567-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:31.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:31 smithi157 ceph-mon[33359]: osdmap e103: 8 total, 8 up, 8 in 2024-03-11T20:54:31.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3017803857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi129-82555-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:31.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi129-82555-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:31.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:31.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi129-82555-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:31.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:31 smithi157 ceph-mon[33359]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T20:54:32.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:32 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:32.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:32 smithi129 ceph-mon[30184]: pgmap v110: 520 pgs: 96 unknown, 424 active+clean; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-03-11T20:54:32.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:32 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:32.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:32 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:32.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:32 smithi129 ceph-mon[30184]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T20:54:32.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:32 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:32.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:32 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:32.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:32 smithi129 ceph-mon[42066]: pgmap v110: 520 pgs: 96 unknown, 424 active+clean; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-03-11T20:54:32.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:32 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:32.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:32 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:32.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:32 smithi129 ceph-mon[42066]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T20:54:32.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:32 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:32.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:32 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:32.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:32 smithi157 ceph-mon[33359]: pgmap v110: 520 pgs: 96 unknown, 424 active+clean; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-03-11T20:54:32.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:32 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:32.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:32 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:32.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:32 smithi157 ceph-mon[33359]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T20:54:32.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:32 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:33.621 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:54:33.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4"}]: dispatch 2024-03-11T20:54:33.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4"}]: dispatch 2024-03-11T20:54:33.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:54:33.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:33.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:33.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4"}]': finished 2024-03-11T20:54:33.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: osdmap e106: 8 total, 8 up, 8 in 2024-03-11T20:54:33.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6"}]: dispatch 2024-03-11T20:54:33.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2877150247' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi129-82567-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:33.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6"}]: dispatch 2024-03-11T20:54:33.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi129-82567-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:33.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/417412581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi129-82555-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:33.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2175729741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:33.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:33.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:33.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:33.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:33.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4"}]: dispatch 2024-03-11T20:54:33.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4"}]: dispatch 2024-03-11T20:54:33.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:54:33.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:33.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:33.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4"}]': finished 2024-03-11T20:54:33.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: osdmap e106: 8 total, 8 up, 8 in 2024-03-11T20:54:33.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6"}]: dispatch 2024-03-11T20:54:33.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2877150247' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi129-82567-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:33.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6"}]: dispatch 2024-03-11T20:54:33.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi129-82567-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:33.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/417412581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi129-82555-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:33.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2175729741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:33.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:33.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:33.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:33.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:33.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4"}]: dispatch 2024-03-11T20:54:33.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4"}]: dispatch 2024-03-11T20:54:33.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:54:33.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:33.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:33.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4"}]': finished 2024-03-11T20:54:33.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: osdmap e106: 8 total, 8 up, 8 in 2024-03-11T20:54:33.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6"}]: dispatch 2024-03-11T20:54:33.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2877150247' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi129-82567-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:33.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6"}]: dispatch 2024-03-11T20:54:33.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi129-82567-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:33.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/417412581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi129-82555-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:33.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2175729741' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:33.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:33.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:33.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:33.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:34.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[42066]: pgmap v113: 680 pgs: 110 creating+peering, 146 unknown, 424 active+clean; 145 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-11T20:54:34.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:34.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:54:34.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6"}]': finished 2024-03-11T20:54:34.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi129-82567-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:34.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/417412581' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi129-82555-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:34.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:34.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:54:34.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[42066]: osdmap e107: 8 total, 8 up, 8 in 2024-03-11T20:54:34.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi129-83101-6", "pool2": "test-rados-api-smithi129-83101-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:54:34.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-9"}]: dispatch 2024-03-11T20:54:34.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi129-83101-6", "pool2": "test-rados-api-smithi129-83101-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:54:34.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-9"}]: dispatch 2024-03-11T20:54:34.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:34.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[42066]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:34.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[30184]: pgmap v113: 680 pgs: 110 creating+peering, 146 unknown, 424 active+clean; 145 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-11T20:54:34.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:34.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:54:34.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6"}]': finished 2024-03-11T20:54:34.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi129-82567-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:34.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/417412581' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi129-82555-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:34.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:34.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:54:34.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[30184]: osdmap e107: 8 total, 8 up, 8 in 2024-03-11T20:54:34.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi129-83101-6", "pool2": "test-rados-api-smithi129-83101-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:54:34.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-9"}]: dispatch 2024-03-11T20:54:34.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi129-83101-6", "pool2": "test-rados-api-smithi129-83101-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:54:34.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-9"}]: dispatch 2024-03-11T20:54:34.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:34.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:34 smithi129 ceph-mon[30184]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:34.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:34 smithi157 ceph-mon[33359]: pgmap v113: 680 pgs: 110 creating+peering, 146 unknown, 424 active+clean; 145 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-11T20:54:34.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:34 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:34.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:34 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:54:34.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:34 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi129-83101-4", "tierpool": "test-rados-api-smithi129-83101-6"}]': finished 2024-03-11T20:54:34.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:34 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi129-82567-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:34.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:34 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/417412581' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi129-82555-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:34.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:34 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:34.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:34 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:54:34.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:34 smithi157 ceph-mon[33359]: osdmap e107: 8 total, 8 up, 8 in 2024-03-11T20:54:34.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:34 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1178483545' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi129-83101-6", "pool2": "test-rados-api-smithi129-83101-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:54:34.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:34 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-9"}]: dispatch 2024-03-11T20:54:34.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:34 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi129-83101-6", "pool2": "test-rados-api-smithi129-83101-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T20:54:34.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:34 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-9"}]: dispatch 2024-03-11T20:54:34.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:34 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:34.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:34 smithi157 ceph-mon[33359]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:35.637 INFO:tasks.workunit.client.0.smithi129.stdout:watch_notify_pp: flushed 2024-03-11T20:54:35.638 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3004 ms) 2024-03-11T20:54:35.638 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-03-11T20:54:35.638 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: trying... 2024-03-11T20:54:35.638 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94296022091792 notify_id 412316860421 notifier_gid 25262 2024-03-11T20:54:35.638 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: timed out 2024-03-11T20:54:35.638 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: flushing 2024-03-11T20:54:35.638 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: flushed 2024-03-11T20:54:35.638 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3005 ms) 2024-03-11T20:54:35.638 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-03-11T20:54:35.638 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: List watches 2024-03-11T20:54:35.638 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify2 2024-03-11T20:54:35.638 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94296022078464 notify_id 425201762308 notifier_gid 25262 2024-03-11T20:54:35.638 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify2 done 2024-03-11T20:54:35.639 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: watch_check 2024-03-11T20:54:35.639 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: unwatch2 2024-03-11T20:54:35.639 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: flushing 2024-03-11T20:54:35.639 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: done 2024-03-11T20:54:35.639 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3006 ms) 2024-03-11T20:54:35.639 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-03-11T20:54:35.639 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: List watches 2024-03-11T20:54:35.639 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify2 2024-03-11T20:54:35.639 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: handle_notify cookie 94296022078464 notify_id 438086664196 notifier_gid 25262 2024-03-11T20:54:35.639 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: notify2 done 2024-03-11T20:54:35.639 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: watch_check 2024-03-11T20:54:35.639 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: unwatch2 2024-03-11T20:54:35.639 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: flushing 2024-03-11T20:54:35.639 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: done 2024-03-11T20:54:35.640 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3006 ms) 2024-03-11T20:54:35.640 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16290 ms total) 2024-03-11T20:54:35.640 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: 2024-03-11T20:54:35.640 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-03-11T20:54:35.640 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (41792 ms total) 2024-03-11T20:54:35.640 INFO:tasks.workunit.client.0.smithi129.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-03-11T20:54:35.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi129-83101-6", "pool2": "test-rados-api-smithi129-83101-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T20:54:35.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-9"}]': finished 2024-03-11T20:54:35.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[30184]: osdmap e108: 8 total, 8 up, 8 in 2024-03-11T20:54:35.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-9", "mode": "writeback"}]: dispatch 2024-03-11T20:54:35.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-9", "mode": "writeback"}]: dispatch 2024-03-11T20:54:35.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:35.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:54:35.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-9", "mode": "writeback"}]': finished 2024-03-11T20:54:35.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[30184]: osdmap e109: 8 total, 8 up, 8 in 2024-03-11T20:54:35.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/542022049' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi129-82555-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:35.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2208948440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi129-82567-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:35.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi129-82555-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:35.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi129-83101-6", "pool2": "test-rados-api-smithi129-83101-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T20:54:35.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-9"}]': finished 2024-03-11T20:54:35.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[42066]: osdmap e108: 8 total, 8 up, 8 in 2024-03-11T20:54:35.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-9", "mode": "writeback"}]: dispatch 2024-03-11T20:54:35.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-9", "mode": "writeback"}]: dispatch 2024-03-11T20:54:35.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:35.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:54:35.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-9", "mode": "writeback"}]': finished 2024-03-11T20:54:35.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[42066]: osdmap e109: 8 total, 8 up, 8 in 2024-03-11T20:54:35.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/542022049' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi129-82555-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:35.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2208948440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi129-82567-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:35.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:35 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi129-82555-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:35.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:35 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi129-83101-6", "pool2": "test-rados-api-smithi129-83101-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T20:54:35.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:35 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-9"}]': finished 2024-03-11T20:54:35.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:35 smithi157 ceph-mon[33359]: osdmap e108: 8 total, 8 up, 8 in 2024-03-11T20:54:35.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-9", "mode": "writeback"}]: dispatch 2024-03-11T20:54:35.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:35 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-9", "mode": "writeback"}]: dispatch 2024-03-11T20:54:35.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:35.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:35 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:54:35.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:35 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-9", "mode": "writeback"}]': finished 2024-03-11T20:54:35.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:35 smithi157 ceph-mon[33359]: osdmap e109: 8 total, 8 up, 8 in 2024-03-11T20:54:35.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/542022049' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi129-82555-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:35.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2208948440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi129-82567-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:35.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:35 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi129-82555-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:36.478 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:54:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:54:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: pgmap v116: 612 pgs: 55 creating+peering, 137 unknown, 420 active+clean; 145 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 op/s 2024-03-11T20:54:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.0"}]: dispatch 2024-03-11T20:54:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.1"}]: dispatch 2024-03-11T20:54:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.2"}]: dispatch 2024-03-11T20:54:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.3"}]: dispatch 2024-03-11T20:54:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.4"}]: dispatch 2024-03-11T20:54:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.5"}]: dispatch 2024-03-11T20:54:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.6"}]: dispatch 2024-03-11T20:54:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.7"}]: dispatch 2024-03-11T20:54:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.8"}]: dispatch 2024-03-11T20:54:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.9"}]: dispatch 2024-03-11T20:54:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: 23.6 deep-scrub starts 2024-03-11T20:54:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: 23.6 deep-scrub ok 2024-03-11T20:54:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: 23.3 deep-scrub starts 2024-03-11T20:54:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: 23.3 deep-scrub ok 2024-03-11T20:54:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2208948440' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi129-82567-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:36.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi129-82555-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:36.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:54:36.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: osdmap e110: 8 total, 8 up, 8 in 2024-03-11T20:54:36.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9"}]: dispatch 2024-03-11T20:54:36.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9"}]: dispatch 2024-03-11T20:54:36.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: pgmap v116: 612 pgs: 55 creating+peering, 137 unknown, 420 active+clean; 145 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 op/s 2024-03-11T20:54:36.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:36.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:36.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.0"}]: dispatch 2024-03-11T20:54:36.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.1"}]: dispatch 2024-03-11T20:54:36.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.2"}]: dispatch 2024-03-11T20:54:36.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.3"}]: dispatch 2024-03-11T20:54:36.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.4"}]: dispatch 2024-03-11T20:54:36.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.5"}]: dispatch 2024-03-11T20:54:36.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.6"}]: dispatch 2024-03-11T20:54:36.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.7"}]: dispatch 2024-03-11T20:54:36.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.8"}]: dispatch 2024-03-11T20:54:36.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.9"}]: dispatch 2024-03-11T20:54:36.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:36.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: 23.6 deep-scrub starts 2024-03-11T20:54:36.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: 23.6 deep-scrub ok 2024-03-11T20:54:36.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: 23.3 deep-scrub starts 2024-03-11T20:54:36.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: 23.3 deep-scrub ok 2024-03-11T20:54:36.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2208948440' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi129-82567-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:36.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi129-82555-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:36.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:54:36.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: osdmap e110: 8 total, 8 up, 8 in 2024-03-11T20:54:36.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9"}]: dispatch 2024-03-11T20:54:36.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9"}]: dispatch 2024-03-11T20:54:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: pgmap v116: 612 pgs: 55 creating+peering, 137 unknown, 420 active+clean; 145 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 op/s 2024-03-11T20:54:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.0"}]: dispatch 2024-03-11T20:54:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.1"}]: dispatch 2024-03-11T20:54:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.2"}]: dispatch 2024-03-11T20:54:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.3"}]: dispatch 2024-03-11T20:54:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.4"}]: dispatch 2024-03-11T20:54:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.5"}]: dispatch 2024-03-11T20:54:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.6"}]: dispatch 2024-03-11T20:54:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.7"}]: dispatch 2024-03-11T20:54:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.8"}]: dispatch 2024-03-11T20:54:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.9"}]: dispatch 2024-03-11T20:54:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: 23.6 deep-scrub starts 2024-03-11T20:54:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: 23.6 deep-scrub ok 2024-03-11T20:54:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: 23.3 deep-scrub starts 2024-03-11T20:54:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: 23.3 deep-scrub ok 2024-03-11T20:54:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2208948440' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi129-82567-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi129-82555-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:36.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:54:36.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: osdmap e110: 8 total, 8 up, 8 in 2024-03-11T20:54:36.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9"}]: dispatch 2024-03-11T20:54:36.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9"}]: dispatch 2024-03-11T20:54:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.0"}]: dispatch 2024-03-11T20:54:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.1"}]: dispatch 2024-03-11T20:54:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.2"}]: dispatch 2024-03-11T20:54:37.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.3"}]: dispatch 2024-03-11T20:54:37.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.4"}]: dispatch 2024-03-11T20:54:37.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.5"}]: dispatch 2024-03-11T20:54:37.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.6"}]: dispatch 2024-03-11T20:54:37.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.7"}]: dispatch 2024-03-11T20:54:37.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.8"}]: dispatch 2024-03-11T20:54:37.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.9"}]: dispatch 2024-03-11T20:54:37.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: 23.4 deep-scrub starts 2024-03-11T20:54:37.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: 23.4 deep-scrub ok 2024-03-11T20:54:37.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: 23.8 deep-scrub starts 2024-03-11T20:54:37.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: 23.8 deep-scrub ok 2024-03-11T20:54:37.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:37.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: 23.7 deep-scrub starts 2024-03-11T20:54:37.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: 23.7 deep-scrub ok 2024-03-11T20:54:37.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: 23.1 deep-scrub starts 2024-03-11T20:54:37.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: 23.1 deep-scrub ok 2024-03-11T20:54:37.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: 23.5 deep-scrub starts 2024-03-11T20:54:37.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: 23.5 deep-scrub ok 2024-03-11T20:54:37.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:37.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:37.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:54:37.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9"}]': finished 2024-03-11T20:54:37.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:37.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.0"}]: dispatch 2024-03-11T20:54:37.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.1"}]: dispatch 2024-03-11T20:54:37.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.2"}]: dispatch 2024-03-11T20:54:37.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.3"}]: dispatch 2024-03-11T20:54:37.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.4"}]: dispatch 2024-03-11T20:54:37.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.5"}]: dispatch 2024-03-11T20:54:37.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.6"}]: dispatch 2024-03-11T20:54:37.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.7"}]: dispatch 2024-03-11T20:54:37.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.8"}]: dispatch 2024-03-11T20:54:37.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.9"}]: dispatch 2024-03-11T20:54:37.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: 23.4 deep-scrub starts 2024-03-11T20:54:37.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: 23.4 deep-scrub ok 2024-03-11T20:54:37.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: 23.8 deep-scrub starts 2024-03-11T20:54:37.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: 23.8 deep-scrub ok 2024-03-11T20:54:37.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:37.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: 23.7 deep-scrub starts 2024-03-11T20:54:37.983 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: 23.7 deep-scrub ok 2024-03-11T20:54:37.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: 23.1 deep-scrub starts 2024-03-11T20:54:37.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: 23.1 deep-scrub ok 2024-03-11T20:54:37.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: 23.5 deep-scrub starts 2024-03-11T20:54:37.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: 23.5 deep-scrub ok 2024-03-11T20:54:37.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:37.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:37.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:54:37.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9"}]': finished 2024-03-11T20:54:37.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:37.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi129-82945-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: osdmap e111: 8 total, 8 up, 8 in 2024-03-11T20:54:37.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2900468096' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:37.984 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi129-82945-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.985 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:37.985 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi129-82945-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.985 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: osdmap e111: 8 total, 8 up, 8 in 2024-03-11T20:54:37.985 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2900468096' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:37.985 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi129-82945-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.985 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:37.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.0"}]: dispatch 2024-03-11T20:54:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.1"}]: dispatch 2024-03-11T20:54:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.2"}]: dispatch 2024-03-11T20:54:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.3"}]: dispatch 2024-03-11T20:54:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.4"}]: dispatch 2024-03-11T20:54:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.5"}]: dispatch 2024-03-11T20:54:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.6"}]: dispatch 2024-03-11T20:54:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.7"}]: dispatch 2024-03-11T20:54:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.8"}]: dispatch 2024-03-11T20:54:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "23.9"}]: dispatch 2024-03-11T20:54:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: 23.4 deep-scrub starts 2024-03-11T20:54:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: 23.4 deep-scrub ok 2024-03-11T20:54:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: 23.8 deep-scrub starts 2024-03-11T20:54:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: 23.8 deep-scrub ok 2024-03-11T20:54:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:37.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: 23.7 deep-scrub starts 2024-03-11T20:54:37.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: 23.7 deep-scrub ok 2024-03-11T20:54:37.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: 23.1 deep-scrub starts 2024-03-11T20:54:37.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: 23.1 deep-scrub ok 2024-03-11T20:54:37.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: 23.5 deep-scrub starts 2024-03-11T20:54:37.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: 23.5 deep-scrub ok 2024-03-11T20:54:37.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:37.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:37.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:54:37.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-9"}]': finished 2024-03-11T20:54:37.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:37.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi129-82945-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: osdmap e111: 8 total, 8 up, 8 in 2024-03-11T20:54:37.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2900468096' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:37.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi129-82945-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:37.989 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:39.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[30184]: pgmap v119: 516 pgs: 32 creating+peering, 4 creating+activating, 480 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s wr, 0 op/s 2024-03-11T20:54:39.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[30184]: 23.9 deep-scrub starts 2024-03-11T20:54:39.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[30184]: 23.9 deep-scrub ok 2024-03-11T20:54:39.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:39.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[30184]: 23.0 deep-scrub starts 2024-03-11T20:54:39.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[30184]: 23.0 deep-scrub ok 2024-03-11T20:54:39.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[30184]: 23.2 deep-scrub starts 2024-03-11T20:54:39.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[30184]: 23.2 deep-scrub ok 2024-03-11T20:54:39.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:39.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[42066]: pgmap v119: 516 pgs: 32 creating+peering, 4 creating+activating, 480 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s wr, 0 op/s 2024-03-11T20:54:39.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[42066]: 23.9 deep-scrub starts 2024-03-11T20:54:39.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[42066]: 23.9 deep-scrub ok 2024-03-11T20:54:39.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:39.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[42066]: 23.0 deep-scrub starts 2024-03-11T20:54:39.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[42066]: 23.0 deep-scrub ok 2024-03-11T20:54:39.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[42066]: 23.2 deep-scrub starts 2024-03-11T20:54:39.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[42066]: 23.2 deep-scrub ok 2024-03-11T20:54:39.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:39.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:39 smithi157 ceph-mon[33359]: pgmap v119: 516 pgs: 32 creating+peering, 4 creating+activating, 480 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s wr, 0 op/s 2024-03-11T20:54:39.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:39 smithi157 ceph-mon[33359]: 23.9 deep-scrub starts 2024-03-11T20:54:39.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:39 smithi157 ceph-mon[33359]: 23.9 deep-scrub ok 2024-03-11T20:54:39.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:39.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:39 smithi157 ceph-mon[33359]: 23.0 deep-scrub starts 2024-03-11T20:54:39.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:39 smithi157 ceph-mon[33359]: 23.0 deep-scrub ok 2024-03-11T20:54:39.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:39 smithi157 ceph-mon[33359]: 23.2 deep-scrub starts 2024-03-11T20:54:39.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:39 smithi157 ceph-mon[33359]: 23.2 deep-scrub ok 2024-03-11T20:54:39.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82826-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: pgmap v122: 548 pgs: 160 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 3 op/s 2024-03-11T20:54:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: osdmap e112: 8 total, 8 up, 8 in 2024-03-11T20:54:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1703402920' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3084747562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi129-82555-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi129-82555-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:54:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi129-82945-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]': finished 2024-03-11T20:54:40.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi129-82555-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:40.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: osdmap e113: 8 total, 8 up, 8 in 2024-03-11T20:54:40.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2900468096' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache", "force_nonempty":""}]: dispatch 2024-03-11T20:54:40.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache", "force_nonempty":""}]: dispatch 2024-03-11T20:54:40.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: pgmap v122: 548 pgs: 160 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 3 op/s 2024-03-11T20:54:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: osdmap e112: 8 total, 8 up, 8 in 2024-03-11T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1703402920' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3084747562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi129-82555-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi129-82555-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi129-82945-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]': finished 2024-03-11T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi129-82555-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: osdmap e113: 8 total, 8 up, 8 in 2024-03-11T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2900468096' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache", "force_nonempty":""}]: dispatch 2024-03-11T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache", "force_nonempty":""}]: dispatch 2024-03-11T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: pgmap v122: 548 pgs: 160 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 3 op/s 2024-03-11T20:54:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: osdmap e112: 8 total, 8 up, 8 in 2024-03-11T20:54:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1703402920' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3084747562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi129-82555-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi129-82555-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:54:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi129-82945-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]': finished 2024-03-11T20:54:40.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi129-82555-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:40.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: osdmap e113: 8 total, 8 up, 8 in 2024-03-11T20:54:40.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2900468096' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache", "force_nonempty":""}]: dispatch 2024-03-11T20:54:40.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache", "force_nonempty":""}]: dispatch 2024-03-11T20:54:40.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:40.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:41.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:54:41.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:54:41.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:54:41.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:41.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache", "force_nonempty":""}]': finished 2024-03-11T20:54:41.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:41.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[30184]: osdmap e114: 8 total, 8 up, 8 in 2024-03-11T20:54:41.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2900468096' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:41.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:41.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:41.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:54:41.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:54:41.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:54:41.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:41.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache", "force_nonempty":""}]': finished 2024-03-11T20:54:41.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:41.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[42066]: osdmap e114: 8 total, 8 up, 8 in 2024-03-11T20:54:41.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2900468096' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:41.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:41.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:41 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:54:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:54:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:54:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:41 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:41 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache", "force_nonempty":""}]': finished 2024-03-11T20:54:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:41 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:41 smithi157 ceph-mon[33359]: osdmap e114: 8 total, 8 up, 8 in 2024-03-11T20:54:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:41 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2900468096' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:41.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:41 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:41.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:41 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:41.662 INFO:tasks.workunit.client.0.smithi129.stdout: misc: Running main() from gmock_main.cc 2024-03-11T20:54:41.663 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-03-11T20:54:41.663 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [----------] Global test environment set-up. 2024-03-11T20:54:41.663 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-03-11T20:54:41.663 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-03-11T20:54:41.663 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.Version (9975 ms) 2024-03-11T20:54:41.663 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-03-11T20:54:41.663 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (4972 ms) 2024-03-11T20:54:41.663 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-03-11T20:54:41.664 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.LongName (4564 ms) 2024-03-11T20:54:41.664 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-03-11T20:54:41.664 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (4690 ms) 2024-03-11T20:54:41.664 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-03-11T20:54:41.664 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3049 ms) 2024-03-11T20:54:41.664 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-03-11T20:54:41.664 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3302 ms) 2024-03-11T20:54:41.664 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-03-11T20:54:41.664 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.Exec (3113 ms) 2024-03-11T20:54:41.664 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-03-11T20:54:41.664 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (2879 ms) 2024-03-11T20:54:41.664 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-03-11T20:54:41.664 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3011 ms) 2024-03-11T20:54:41.664 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-03-11T20:54:41.665 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.BigObject (2993 ms) 2024-03-11T20:54:41.665 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-03-11T20:54:41.665 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2250 ms) 2024-03-11T20:54:41.665 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-03-11T20:54:41.665 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (2764 ms) 2024-03-11T20:54:41.665 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [----------] 12 tests from NeoRadosMisc (47562 ms total) 2024-03-11T20:54:41.665 INFO:tasks.workunit.client.0.smithi129.stdout: misc: 2024-03-11T20:54:41.665 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [----------] Global test environment tear-down 2024-03-11T20:54:41.665 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [==========] 12 tests from 1 test suite ran. (47562 ms total) 2024-03-11T20:54:41.665 INFO:tasks.workunit.client.0.smithi129.stdout: misc: [ PASSED ] 12 tests. 2024-03-11T20:54:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[30184]: pgmap v125: 588 pgs: 200 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail 2024-03-11T20:54:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[30184]: osdmap e115: 8 total, 8 up, 8 in 2024-03-11T20:54:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2176450140' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi129-82555-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi129-82555-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2900468096' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache"}]: dispatch 2024-03-11T20:54:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache"}]: dispatch 2024-03-11T20:54:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:42.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:42.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:42.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[42066]: pgmap v125: 588 pgs: 200 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail 2024-03-11T20:54:42.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:42.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[42066]: osdmap e115: 8 total, 8 up, 8 in 2024-03-11T20:54:42.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2176450140' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi129-82555-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:42.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi129-82555-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:42.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2900468096' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache"}]: dispatch 2024-03-11T20:54:42.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache"}]: dispatch 2024-03-11T20:54:42.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:42.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:42.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:42 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:42.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:42 smithi157 ceph-mon[33359]: pgmap v125: 588 pgs: 200 unknown, 32 creating+peering, 356 active+clean; 217 MiB data, 1017 MiB used, 714 GiB / 715 GiB avail 2024-03-11T20:54:42.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:42 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82826-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:42.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:42 smithi157 ceph-mon[33359]: osdmap e115: 8 total, 8 up, 8 in 2024-03-11T20:54:42.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2176450140' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi129-82555-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:42.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:42 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi129-82555-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:42.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2900468096' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache"}]: dispatch 2024-03-11T20:54:42.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:42 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache"}]: dispatch 2024-03-11T20:54:42.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:42.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:42.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:42 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:43.695 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:54:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:54:43.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi129-82555-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:43.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache"}]': finished 2024-03-11T20:54:43.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:54:43.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[30184]: osdmap e116: 8 total, 8 up, 8 in 2024-03-11T20:54:43.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/285903083' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi129-82567-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:43.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-11"}]: dispatch 2024-03-11T20:54:43.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-11"}]: dispatch 2024-03-11T20:54:43.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:43.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/285903083' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi129-82567-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:43.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-11"}]': finished 2024-03-11T20:54:43.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[30184]: osdmap e117: 8 total, 8 up, 8 in 2024-03-11T20:54:43.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-11", "mode": "writeback"}]: dispatch 2024-03-11T20:54:43.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-11", "mode": "writeback"}]: dispatch 2024-03-11T20:54:43.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi129-82555-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:43.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache"}]': finished 2024-03-11T20:54:43.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:54:43.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[42066]: osdmap e116: 8 total, 8 up, 8 in 2024-03-11T20:54:43.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/285903083' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi129-82567-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:43.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-11"}]: dispatch 2024-03-11T20:54:43.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-11"}]: dispatch 2024-03-11T20:54:43.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:43.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/285903083' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi129-82567-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:43.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-11"}]': finished 2024-03-11T20:54:43.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[42066]: osdmap e117: 8 total, 8 up, 8 in 2024-03-11T20:54:43.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-11", "mode": "writeback"}]: dispatch 2024-03-11T20:54:43.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-11", "mode": "writeback"}]: dispatch 2024-03-11T20:54:43.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi129-82555-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:43.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82826-10", "tierpool":"test-rados-api-smithi129-82826-10-cache"}]': finished 2024-03-11T20:54:43.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:54:43.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:43 smithi157 ceph-mon[33359]: osdmap e116: 8 total, 8 up, 8 in 2024-03-11T20:54:43.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/285903083' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi129-82567-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:43.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-11"}]: dispatch 2024-03-11T20:54:43.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-11"}]: dispatch 2024-03-11T20:54:43.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:43.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/285903083' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi129-82567-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:43.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-11"}]': finished 2024-03-11T20:54:43.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:43 smithi157 ceph-mon[33359]: osdmap e117: 8 total, 8 up, 8 in 2024-03-11T20:54:43.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-11", "mode": "writeback"}]: dispatch 2024-03-11T20:54:43.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-11", "mode": "writeback"}]: dispatch 2024-03-11T20:54:44.653 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: Running main() from gmock_main.cc 2024-03-11T20:54:44.653 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-03-11T20:54:44.653 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [----------] Global test environment set-up. 2024-03-11T20:54:44.653 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-03-11T20:54:44.653 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-03-11T20:54:44.653 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolList (10320 ms) 2024-03-11T20:54:44.653 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-03-11T20:54:44.653 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (6254 ms) 2024-03-11T20:54:44.653 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-03-11T20:54:44.653 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3831 ms) 2024-03-11T20:54:44.653 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-03-11T20:54:44.653 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (4147 ms) 2024-03-11T20:54:44.654 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-03-11T20:54:44.654 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3077 ms) 2024-03-11T20:54:44.654 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-03-11T20:54:44.654 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5372 ms) 2024-03-11T20:54:44.654 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-03-11T20:54:44.654 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5149 ms) 2024-03-11T20:54:44.654 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-03-11T20:54:44.654 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (4738 ms) 2024-03-11T20:54:44.654 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-03-11T20:54:44.654 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8010 ms) 2024-03-11T20:54:44.654 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [----------] 9 tests from LibRadosPools (50898 ms total) 2024-03-11T20:54:44.654 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: 2024-03-11T20:54:44.654 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [----------] Global test environment tear-down 2024-03-11T20:54:44.654 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (50898 ms total) 2024-03-11T20:54:44.655 INFO:tasks.workunit.client.0.smithi129.stdout: api_pool: [ PASSED ] 9 tests. 2024-03-11T20:54:44.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[42066]: pgmap v128: 556 pgs: 32 unknown, 64 creating+peering, 460 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 259 KiB/s wr, 5 op/s 2024-03-11T20:54:44.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:44.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:44.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[42066]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:44.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:54:44.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-11", "mode": "writeback"}]': finished 2024-03-11T20:54:44.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[42066]: osdmap e118: 8 total, 8 up, 8 in 2024-03-11T20:54:44.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/440972941' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi129-82555-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:44.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi129-82555-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:44.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[30184]: pgmap v128: 556 pgs: 32 unknown, 64 creating+peering, 460 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 259 KiB/s wr, 5 op/s 2024-03-11T20:54:44.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:44.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:44.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[30184]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:44.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:54:44.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-11", "mode": "writeback"}]': finished 2024-03-11T20:54:44.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[30184]: osdmap e118: 8 total, 8 up, 8 in 2024-03-11T20:54:44.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/440972941' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi129-82555-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:44.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:44 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi129-82555-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:44.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:44 smithi157 ceph-mon[33359]: pgmap v128: 556 pgs: 32 unknown, 64 creating+peering, 460 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 259 KiB/s wr, 5 op/s 2024-03-11T20:54:44.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:44 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:44.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:44.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:44 smithi157 ceph-mon[33359]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:44.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:44 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:54:44.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:44 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-11", "mode": "writeback"}]': finished 2024-03-11T20:54:44.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:44 smithi157 ceph-mon[33359]: osdmap e118: 8 total, 8 up, 8 in 2024-03-11T20:54:44.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/440972941' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi129-82555-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:44.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:44 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi129-82555-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:46.228 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:54:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:54:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:54:46.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:46.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:45 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi129-82555-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:46.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:45 smithi129 ceph-mon[30184]: osdmap e119: 8 total, 8 up, 8 in 2024-03-11T20:54:46.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2122719862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi129-82567-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:46.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:46.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:45 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi129-82555-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:46.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:45 smithi129 ceph-mon[42066]: osdmap e119: 8 total, 8 up, 8 in 2024-03-11T20:54:46.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2122719862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi129-82567-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:46.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:46.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:45 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi129-82555-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:46.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:45 smithi157 ceph-mon[33359]: osdmap e119: 8 total, 8 up, 8 in 2024-03-11T20:54:46.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2122719862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi129-82567-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:47.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[42066]: pgmap v131: 524 pgs: 128 unknown, 396 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 259 KiB/s wr, 5 op/s 2024-03-11T20:54:47.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:47.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:47.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:47.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2122719862' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi129-82567-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:47.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:54:47.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[42066]: osdmap e120: 8 total, 8 up, 8 in 2024-03-11T20:54:47.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11"}]: dispatch 2024-03-11T20:54:47.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11"}]: dispatch 2024-03-11T20:54:47.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:47.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[30184]: pgmap v131: 524 pgs: 128 unknown, 396 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 259 KiB/s wr, 5 op/s 2024-03-11T20:54:47.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:47.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:47.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:47.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2122719862' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi129-82567-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:47.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:54:47.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[30184]: osdmap e120: 8 total, 8 up, 8 in 2024-03-11T20:54:47.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11"}]: dispatch 2024-03-11T20:54:47.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11"}]: dispatch 2024-03-11T20:54:47.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:47.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:47 smithi157 ceph-mon[33359]: pgmap v131: 524 pgs: 128 unknown, 396 active+clean; 218 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 259 KiB/s wr, 5 op/s 2024-03-11T20:54:47.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:47.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:47.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:47 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:54:47.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2122719862' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi129-82567-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:47.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:47 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:54:47.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:47 smithi157 ceph-mon[33359]: osdmap e120: 8 total, 8 up, 8 in 2024-03-11T20:54:47.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11"}]: dispatch 2024-03-11T20:54:47.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:47 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11"}]: dispatch 2024-03-11T20:54:47.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:48.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:48 smithi129 ceph-mon[42066]: pgmap v134: 492 pgs: 32 creating+peering, 460 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-11T20:54:48.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:48 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:54:48.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:48 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11"}]': finished 2024-03-11T20:54:48.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:48 smithi129 ceph-mon[42066]: osdmap e121: 8 total, 8 up, 8 in 2024-03-11T20:54:48.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:48 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:48.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/149630651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi129-82555-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:48.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:48 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi129-82555-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:48.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:48.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:48 smithi129 ceph-mon[30184]: pgmap v134: 492 pgs: 32 creating+peering, 460 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-11T20:54:48.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:48 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:54:48.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:48 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11"}]': finished 2024-03-11T20:54:48.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:48 smithi129 ceph-mon[30184]: osdmap e121: 8 total, 8 up, 8 in 2024-03-11T20:54:48.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:48 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:48.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/149630651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi129-82555-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:48.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:48 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi129-82555-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:48.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:48.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:48 smithi157 ceph-mon[33359]: pgmap v134: 492 pgs: 32 creating+peering, 460 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-11T20:54:48.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:48 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:54:48.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:48 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-11"}]': finished 2024-03-11T20:54:48.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:48 smithi157 ceph-mon[33359]: osdmap e121: 8 total, 8 up, 8 in 2024-03-11T20:54:48.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:48 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:54:48.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/149630651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi129-82555-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:48.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:48 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi129-82555-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:48.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:48.667 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: Running main() from gmock_main.cc 2024-03-11T20:54:48.667 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-03-11T20:54:48.667 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [----------] Global test environment set-up. 2024-03-11T20:54:48.667 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-03-11T20:54:48.667 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-03-11T20:54:48.668 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (9980 ms) 2024-03-11T20:54:48.668 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-03-11T20:54:48.668 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (6242 ms) 2024-03-11T20:54:48.668 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-03-11T20:54:48.668 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (4892 ms) 2024-03-11T20:54:48.668 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-03-11T20:54:48.668 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3080 ms) 2024-03-11T20:54:48.668 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-03-11T20:54:48.668 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3043 ms) 2024-03-11T20:54:48.668 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-03-11T20:54:48.668 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3309 ms) 2024-03-11T20:54:48.668 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-03-11T20:54:48.668 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3115 ms) 2024-03-11T20:54:48.668 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-03-11T20:54:48.668 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (2867 ms) 2024-03-11T20:54:48.669 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-03-11T20:54:48.669 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3032 ms) 2024-03-11T20:54:48.669 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-03-11T20:54:48.669 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (2978 ms) 2024-03-11T20:54:48.669 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-03-11T20:54:48.669 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3010 ms) 2024-03-11T20:54:48.669 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-03-11T20:54:48.669 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3022 ms) 2024-03-11T20:54:48.669 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-03-11T20:54:48.669 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (2979 ms) 2024-03-11T20:54:48.669 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-03-11T20:54:48.669 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3010 ms) 2024-03-11T20:54:48.669 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (54559 ms total) 2024-03-11T20:54:48.669 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: 2024-03-11T20:54:48.669 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [----------] Global test environment tear-down 2024-03-11T20:54:48.670 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (54559 ms total) 2024-03-11T20:54:48.670 INFO:tasks.workunit.client.0.smithi129.stdout: read_operations: [ PASSED ] 14 tests. 2024-03-11T20:54:49.664 INFO:tasks.workunit.client.0.smithi129.stdout: io: Running main() from gmock_main.cc 2024-03-11T20:54:49.664 INFO:tasks.workunit.client.0.smithi129.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-03-11T20:54:49.664 INFO:tasks.workunit.client.0.smithi129.stdout: io: [----------] Global test environment set-up. 2024-03-11T20:54:49.664 INFO:tasks.workunit.client.0.smithi129.stdout: io: [----------] 14 tests from NeoRadosIo 2024-03-11T20:54:49.664 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-03-11T20:54:49.664 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.Limits (10953 ms) 2024-03-11T20:54:49.664 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-03-11T20:54:49.664 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (7559 ms) 2024-03-11T20:54:49.664 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-03-11T20:54:49.664 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.ReadOp (3660 ms) 2024-03-11T20:54:49.664 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-03-11T20:54:49.664 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.SparseRead (3122 ms) 2024-03-11T20:54:49.665 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-03-11T20:54:49.665 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3313 ms) 2024-03-11T20:54:49.665 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-03-11T20:54:49.665 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3106 ms) 2024-03-11T20:54:49.665 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-03-11T20:54:49.665 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3013 ms) 2024-03-11T20:54:49.665 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-03-11T20:54:49.665 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (2878 ms) 2024-03-11T20:54:49.665 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-03-11T20:54:49.665 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (2992 ms) 2024-03-11T20:54:49.665 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-03-11T20:54:49.665 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.Trunc (3004 ms) 2024-03-11T20:54:49.665 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-03-11T20:54:49.665 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.Remove (3006 ms) 2024-03-11T20:54:49.666 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-03-11T20:54:49.666 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3014 ms) 2024-03-11T20:54:49.666 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-03-11T20:54:49.666 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.RmXattr (2988 ms) 2024-03-11T20:54:49.666 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-03-11T20:54:49.666 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3005 ms) 2024-03-11T20:54:49.666 INFO:tasks.workunit.client.0.smithi129.stdout: io: [----------] 14 tests from NeoRadosIo (55613 ms total) 2024-03-11T20:54:49.666 INFO:tasks.workunit.client.0.smithi129.stdout: io: 2024-03-11T20:54:49.666 INFO:tasks.workunit.client.0.smithi129.stdout: io: [----------] Global test environment tear-down 2024-03-11T20:54:49.666 INFO:tasks.workunit.client.0.smithi129.stdout: io: [==========] 14 tests from 1 test suite ran. (55613 ms total) 2024-03-11T20:54:49.666 INFO:tasks.workunit.client.0.smithi129.stdout: io: [ PASSED ] 14 tests. 2024-03-11T20:54:49.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi129-82555-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:49.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[42066]: osdmap e122: 8 total, 8 up, 8 in 2024-03-11T20:54:49.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3224679568' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi129-82567-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:49.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi129-82567-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:49.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:49.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi129-82567-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:49.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[42066]: osdmap e123: 8 total, 8 up, 8 in 2024-03-11T20:54:49.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:49.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:49.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi129-82555-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:49.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[30184]: osdmap e122: 8 total, 8 up, 8 in 2024-03-11T20:54:49.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3224679568' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi129-82567-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:49.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi129-82567-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:49.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:49.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi129-82567-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:49.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[30184]: osdmap e123: 8 total, 8 up, 8 in 2024-03-11T20:54:49.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:49.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:49.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi129-82555-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:49.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:49 smithi157 ceph-mon[33359]: osdmap e122: 8 total, 8 up, 8 in 2024-03-11T20:54:49.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3224679568' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi129-82567-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:49.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi129-82567-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:49.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:49.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi129-82567-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:49.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:49 smithi157 ceph-mon[33359]: osdmap e123: 8 total, 8 up, 8 in 2024-03-11T20:54:49.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:49.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:50.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[30184]: pgmap v137: 492 pgs: 96 unknown, 396 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 5.5 KiB/s wr, 8 op/s 2024-03-11T20:54:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[30184]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]': finished 2024-03-11T20:54:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[30184]: osdmap e124: 8 total, 8 up, 8 in 2024-03-11T20:54:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3013157068' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi129-82555-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[42066]: pgmap v137: 492 pgs: 96 unknown, 396 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 5.5 KiB/s wr, 8 op/s 2024-03-11T20:54:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[42066]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]': finished 2024-03-11T20:54:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[42066]: osdmap e124: 8 total, 8 up, 8 in 2024-03-11T20:54:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3013157068' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi129-82555-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:50.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:50 smithi157 ceph-mon[33359]: pgmap v137: 492 pgs: 96 unknown, 396 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 5.5 KiB/s wr, 8 op/s 2024-03-11T20:54:50.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:50 smithi157 ceph-mon[33359]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:50.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:50.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:50.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:50.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:50.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi129-82945-10"}]': finished 2024-03-11T20:54:50.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/258996832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:50.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:50 smithi157 ceph-mon[33359]: osdmap e124: 8 total, 8 up, 8 in 2024-03-11T20:54:50.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi129-82945-10"}]: dispatch 2024-03-11T20:54:50.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3013157068' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi129-82555-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:51.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:51.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:51 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi129-82945-10"}]': finished 2024-03-11T20:54:51.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3013157068' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi129-82555-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:51.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:51 smithi129 ceph-mon[30184]: osdmap e125: 8 total, 8 up, 8 in 2024-03-11T20:54:51.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3962931091' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi129-82567-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:51.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:51.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:51 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi129-82945-10"}]': finished 2024-03-11T20:54:51.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3013157068' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi129-82555-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:51.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:51 smithi129 ceph-mon[42066]: osdmap e125: 8 total, 8 up, 8 in 2024-03-11T20:54:51.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3962931091' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi129-82567-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:51.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:51.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:51 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi129-82945-10"}]': finished 2024-03-11T20:54:51.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3013157068' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi129-82555-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:51.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:51 smithi157 ceph-mon[33359]: osdmap e125: 8 total, 8 up, 8 in 2024-03-11T20:54:51.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3962931091' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi129-82567-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:52.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[30184]: pgmap v140: 420 pgs: 64 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:54:52.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:52.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:52.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:52.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3962931091' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi129-82567-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:52.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:52.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[30184]: osdmap e126: 8 total, 8 up, 8 in 2024-03-11T20:54:52.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[42066]: pgmap v140: 420 pgs: 64 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:54:52.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:52.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:52.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:52.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3962931091' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi129-82567-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:52.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:52.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[42066]: osdmap e126: 8 total, 8 up, 8 in 2024-03-11T20:54:52.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:52 smithi157 ceph-mon[33359]: pgmap v140: 420 pgs: 64 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:54:52.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:52.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:54:52.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:52.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3962931091' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi129-82567-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:52.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:54:52.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:52 smithi157 ceph-mon[33359]: osdmap e126: 8 total, 8 up, 8 in 2024-03-11T20:54:52.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:52.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:54:53.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:54:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:54:53.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:53 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:54.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:53 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:54.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:53 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:55.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[30184]: pgmap v143: 452 pgs: 64 creating+peering, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 30 KiB/s wr, 30 op/s 2024-03-11T20:54:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[30184]: osdmap e127: 8 total, 8 up, 8 in 2024-03-11T20:54:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2274071163' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi129-82555-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi129-82555-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]': finished 2024-03-11T20:54:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi129-82555-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[30184]: osdmap e128: 8 total, 8 up, 8 in 2024-03-11T20:54:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4092346102' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi129-82567-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:55.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[42066]: pgmap v143: 452 pgs: 64 creating+peering, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 30 KiB/s wr, 30 op/s 2024-03-11T20:54:55.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:55.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:55.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[42066]: osdmap e127: 8 total, 8 up, 8 in 2024-03-11T20:54:55.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2274071163' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi129-82555-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:55.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi129-82555-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:55.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]': finished 2024-03-11T20:54:55.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi129-82555-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:55.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[42066]: osdmap e128: 8 total, 8 up, 8 in 2024-03-11T20:54:55.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4092346102' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi129-82567-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:55.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:55.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:55.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:55.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:54 smithi157 ceph-mon[33359]: pgmap v143: 452 pgs: 64 creating+peering, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 30 KiB/s wr, 30 op/s 2024-03-11T20:54:55.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:54 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:54:55.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:55.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:54 smithi157 ceph-mon[33359]: osdmap e127: 8 total, 8 up, 8 in 2024-03-11T20:54:55.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2274071163' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi129-82555-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:55.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi129-82555-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:55.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi129-82945-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]': finished 2024-03-11T20:54:55.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi129-82555-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:55.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:54 smithi157 ceph-mon[33359]: osdmap e128: 8 total, 8 up, 8 in 2024-03-11T20:54:55.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4092346102' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi129-82567-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:55.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:55.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:55.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:54:56.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-03-11T20:54:56.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] Global test environment set-up. 2024-03-11T20:54:56.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-03-11T20:54:56.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-03-11T20:54:56.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-03-11T20:54:56.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-11T20:54:56.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2024-03-11T20:54:56.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-03-11T20:54:56.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-03-11T20:54:56.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-03-11T20:54:56.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2024-03-11T20:53:53.803+0000 7f16b0b58780 -1 failed for service _ceph-mon._tcp 2024-03-11T20:54:56.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2024-03-11T20:53:53.803+0000 7f16b0b58780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-03-11T20:54:56.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (58 ms) 2024-03-11T20:54:56.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-03-11T20:54:56.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2019 ms) 2024-03-11T20:54:56.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2077 ms total) 2024-03-11T20:54:56.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2024-03-11T20:54:56.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-03-11T20:54:56.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-03-11T20:54:56.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x5582546ed600 2024-03-11T20:54:56.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a2dc20 2024-03-11T20:54:56.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a2e5c0 2024-03-11T20:54:56.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a2ebc0 2024-03-11T20:54:56.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a2eab0 2024-03-11T20:54:56.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a2f110 2024-03-11T20:54:56.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254996810 2024-03-11T20:54:56.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a2f9b0 2024-03-11T20:54:56.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a2fec0 2024-03-11T20:54:56.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a30370 2024-03-11T20:54:56.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a30020 2024-03-11T20:54:56.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a30150 2024-03-11T20:54:56.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a30260 2024-03-11T20:54:56.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a30990 2024-03-11T20:54:56.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a2fac0 2024-03-11T20:54:56.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a2fbf0 2024-03-11T20:54:56.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a2fd00 2024-03-11T20:54:56.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a30e60 2024-03-11T20:54:56.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a304e0 2024-03-11T20:54:56.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a30aa0 2024-03-11T20:54:56.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a30bd0 2024-03-11T20:54:56.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a30ce0 2024-03-11T20:54:56.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a31420 2024-03-11T20:54:56.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a31310 2024-03-11T20:54:56.094 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a30630 2024-03-11T20:54:56.094 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a30740 2024-03-11T20:54:56.094 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a30850 2024-03-11T20:54:56.094 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a31a20 2024-03-11T20:54:56.094 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a31530 2024-03-11T20:54:56.094 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a30f70 2024-03-11T20:54:56.094 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a310a0 2024-03-11T20:54:56.094 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a311d0 2024-03-11T20:54:56.094 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a31fe0 2024-03-11T20:54:56.094 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a31b30 2024-03-11T20:54:56.094 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a31c60 2024-03-11T20:54:56.094 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a31d90 2024-03-11T20:54:56.094 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a31ec0 2024-03-11T20:54:56.095 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a325c0 2024-03-11T20:54:56.095 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a324b0 2024-03-11T20:54:56.095 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a31680 2024-03-11T20:54:56.095 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a317b0 2024-03-11T20:54:56.095 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a326d0 2024-03-11T20:54:56.095 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a327e0 2024-03-11T20:54:56.095 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a32de0 2024-03-11T20:54:56.095 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a32930 2024-03-11T20:54:56.095 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a32a60 2024-03-11T20:54:56.095 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a32bb0 2024-03-11T20:54:56.095 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a32ef0 2024-03-11T20:54:56.095 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a333a0 2024-03-11T20:54:56.095 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a334b0 2024-03-11T20:54:56.095 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a33a70 2024-03-11T20:54:56.228 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:54:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:54:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:54:56.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:55 smithi129 ceph-mon[30184]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:56.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4092346102' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi129-82567-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:56.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:54:56.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:55 smithi129 ceph-mon[30184]: osdmap e129: 8 total, 8 up, 8 in 2024-03-11T20:54:56.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-13"}]: dispatch 2024-03-11T20:54:56.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-13"}]: dispatch 2024-03-11T20:54:56.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:55 smithi129 ceph-mon[42066]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:56.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4092346102' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi129-82567-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:56.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:54:56.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:55 smithi129 ceph-mon[42066]: osdmap e129: 8 total, 8 up, 8 in 2024-03-11T20:54:56.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-13"}]: dispatch 2024-03-11T20:54:56.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-13"}]: dispatch 2024-03-11T20:54:56.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:55 smithi157 ceph-mon[33359]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:54:56.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4092346102' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi129-82567-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:56.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:54:56.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:55 smithi157 ceph-mon[33359]: osdmap e129: 8 total, 8 up, 8 in 2024-03-11T20:54:56.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-13"}]: dispatch 2024-03-11T20:54:56.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-13"}]: dispatch 2024-03-11T20:54:57.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[30184]: pgmap v146: 492 pgs: 72 unknown, 32 creating+peering, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 35 KiB/s wr, 36 op/s 2024-03-11T20:54:57.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:57.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-13"}]': finished 2024-03-11T20:54:57.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[30184]: osdmap e130: 8 total, 8 up, 8 in 2024-03-11T20:54:57.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2396413867' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi129-82555-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:57.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-13", "mode": "writeback"}]: dispatch 2024-03-11T20:54:57.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi129-82555-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:57.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-13", "mode": "writeback"}]: dispatch 2024-03-11T20:54:57.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:57.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[42066]: pgmap v146: 492 pgs: 72 unknown, 32 creating+peering, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 35 KiB/s wr, 36 op/s 2024-03-11T20:54:57.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:57.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-13"}]': finished 2024-03-11T20:54:57.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[42066]: osdmap e130: 8 total, 8 up, 8 in 2024-03-11T20:54:57.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2396413867' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi129-82555-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:57.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-13", "mode": "writeback"}]: dispatch 2024-03-11T20:54:57.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi129-82555-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:57.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-13", "mode": "writeback"}]: dispatch 2024-03-11T20:54:57.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:57.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:57 smithi157 ceph-mon[33359]: pgmap v146: 492 pgs: 72 unknown, 32 creating+peering, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 35 KiB/s wr, 36 op/s 2024-03-11T20:54:57.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:57.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-13"}]': finished 2024-03-11T20:54:57.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:57 smithi157 ceph-mon[33359]: osdmap e130: 8 total, 8 up, 8 in 2024-03-11T20:54:57.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2396413867' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi129-82555-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:57.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-13", "mode": "writeback"}]: dispatch 2024-03-11T20:54:57.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi129-82555-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:57.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-13", "mode": "writeback"}]: dispatch 2024-03-11T20:54:57.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:58.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:58 smithi129 ceph-mon[30184]: pgmap v149: 428 pgs: 32 creating+peering, 396 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 94 KiB/s wr, 91 op/s 2024-03-11T20:54:58.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:58 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:54:58.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:58 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi129-82555-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:58.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:58 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-13", "mode": "writeback"}]': finished 2024-03-11T20:54:58.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:58 smithi129 ceph-mon[30184]: osdmap e131: 8 total, 8 up, 8 in 2024-03-11T20:54:58.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1291547395' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi129-82567-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:58.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:58 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi129-82567-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:58.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:58.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:58 smithi129 ceph-mon[42066]: pgmap v149: 428 pgs: 32 creating+peering, 396 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 94 KiB/s wr, 91 op/s 2024-03-11T20:54:58.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:58 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:54:58.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:58 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi129-82555-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:58.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:58 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-13", "mode": "writeback"}]': finished 2024-03-11T20:54:58.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:58 smithi129 ceph-mon[42066]: osdmap e131: 8 total, 8 up, 8 in 2024-03-11T20:54:58.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1291547395' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi129-82567-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:58.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:58 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi129-82567-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:58.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:58.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:58 smithi157 ceph-mon[33359]: pgmap v149: 428 pgs: 32 creating+peering, 396 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 94 KiB/s wr, 91 op/s 2024-03-11T20:54:58.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:58 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:54:58.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:58 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi129-82555-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:58.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:58 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-13", "mode": "writeback"}]': finished 2024-03-11T20:54:58.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:58 smithi157 ceph-mon[33359]: osdmap e131: 8 total, 8 up, 8 in 2024-03-11T20:54:58.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1291547395' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi129-82567-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:58.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:58 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi129-82567-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:54:58.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:59.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.0"}]: dispatch 2024-03-11T20:54:59.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.0"}]: dispatch 2024-03-11T20:54:59.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.1"}]: dispatch 2024-03-11T20:54:59.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.1"}]: dispatch 2024-03-11T20:54:59.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.2"}]: dispatch 2024-03-11T20:54:59.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.2"}]: dispatch 2024-03-11T20:54:59.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.3"}]: dispatch 2024-03-11T20:54:59.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.3"}]: dispatch 2024-03-11T20:54:59.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.4"}]: dispatch 2024-03-11T20:54:59.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.4"}]: dispatch 2024-03-11T20:54:59.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.5"}]: dispatch 2024-03-11T20:54:59.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.5"}]: dispatch 2024-03-11T20:54:59.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.6"}]: dispatch 2024-03-11T20:54:59.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.6"}]: dispatch 2024-03-11T20:54:59.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.7"}]: dispatch 2024-03-11T20:54:59.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.7"}]: dispatch 2024-03-11T20:54:59.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.8"}]: dispatch 2024-03-11T20:54:59.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.8"}]: dispatch 2024-03-11T20:54:59.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.9"}]: dispatch 2024-03-11T20:54:59.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.9"}]: dispatch 2024-03-11T20:54:59.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: 174.4 deep-scrub starts 2024-03-11T20:54:59.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: 174.4 deep-scrub ok 2024-03-11T20:54:59.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: 174.6 scrub starts 2024-03-11T20:54:59.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: 174.6 scrub ok 2024-03-11T20:54:59.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi129-82567-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:59.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: osdmap e132: 8 total, 8 up, 8 in 2024-03-11T20:54:59.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:59.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.0"}]: dispatch 2024-03-11T20:54:59.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.0"}]: dispatch 2024-03-11T20:54:59.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.1"}]: dispatch 2024-03-11T20:54:59.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.1"}]: dispatch 2024-03-11T20:54:59.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.2"}]: dispatch 2024-03-11T20:54:59.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.2"}]: dispatch 2024-03-11T20:54:59.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.3"}]: dispatch 2024-03-11T20:54:59.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.3"}]: dispatch 2024-03-11T20:54:59.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.4"}]: dispatch 2024-03-11T20:54:59.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.4"}]: dispatch 2024-03-11T20:54:59.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.5"}]: dispatch 2024-03-11T20:54:59.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.5"}]: dispatch 2024-03-11T20:54:59.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.6"}]: dispatch 2024-03-11T20:54:59.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.6"}]: dispatch 2024-03-11T20:54:59.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.7"}]: dispatch 2024-03-11T20:54:59.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.7"}]: dispatch 2024-03-11T20:54:59.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.8"}]: dispatch 2024-03-11T20:54:59.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.8"}]: dispatch 2024-03-11T20:54:59.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.9"}]: dispatch 2024-03-11T20:54:59.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.9"}]: dispatch 2024-03-11T20:54:59.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: 174.4 deep-scrub starts 2024-03-11T20:54:59.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: 174.4 deep-scrub ok 2024-03-11T20:54:59.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: 174.6 scrub starts 2024-03-11T20:54:59.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: 174.6 scrub ok 2024-03-11T20:54:59.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi129-82567-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:59.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: osdmap e132: 8 total, 8 up, 8 in 2024-03-11T20:54:59.483 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:54:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:54:59.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.0"}]: dispatch 2024-03-11T20:54:59.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.0"}]: dispatch 2024-03-11T20:54:59.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.1"}]: dispatch 2024-03-11T20:54:59.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.1"}]: dispatch 2024-03-11T20:54:59.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.2"}]: dispatch 2024-03-11T20:54:59.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.2"}]: dispatch 2024-03-11T20:54:59.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.3"}]: dispatch 2024-03-11T20:54:59.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.3"}]: dispatch 2024-03-11T20:54:59.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.4"}]: dispatch 2024-03-11T20:54:59.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.4"}]: dispatch 2024-03-11T20:54:59.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.5"}]: dispatch 2024-03-11T20:54:59.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.5"}]: dispatch 2024-03-11T20:54:59.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.6"}]: dispatch 2024-03-11T20:54:59.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.6"}]: dispatch 2024-03-11T20:54:59.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.7"}]: dispatch 2024-03-11T20:54:59.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.7"}]: dispatch 2024-03-11T20:54:59.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.8"}]: dispatch 2024-03-11T20:54:59.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.8"}]: dispatch 2024-03-11T20:54:59.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "174.9"}]: dispatch 2024-03-11T20:54:59.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "174.9"}]: dispatch 2024-03-11T20:54:59.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: 174.4 deep-scrub starts 2024-03-11T20:54:59.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: 174.4 deep-scrub ok 2024-03-11T20:54:59.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: 174.6 scrub starts 2024-03-11T20:54:59.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: 174.6 scrub ok 2024-03-11T20:54:59.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi129-82567-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:54:59.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: osdmap e132: 8 total, 8 up, 8 in 2024-03-11T20:54:59.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:54:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:00.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: 174.5 scrub starts 2024-03-11T20:55:00.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: 174.5 scrub ok 2024-03-11T20:55:00.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: pgmap v152: 460 pgs: 64 unknown, 396 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 94 KiB/s wr, 91 op/s 2024-03-11T20:55:00.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: 174.1 scrub starts 2024-03-11T20:55:00.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: 174.1 scrub ok 2024-03-11T20:55:00.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: 174.9 scrub starts 2024-03-11T20:55:00.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: 174.9 scrub ok 2024-03-11T20:55:00.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: 174.8 scrub starts 2024-03-11T20:55:00.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: 174.8 scrub ok 2024-03-11T20:55:00.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: 174.7 deep-scrub starts 2024-03-11T20:55:00.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: 174.7 deep-scrub ok 2024-03-11T20:55:00.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: 174.2 scrub starts 2024-03-11T20:55:00.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: 174.2 scrub ok 2024-03-11T20:55:00.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: 174.0 scrub starts 2024-03-11T20:55:00.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: 174.0 scrub ok 2024-03-11T20:55:00.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: osdmap e133: 8 total, 8 up, 8 in 2024-03-11T20:55:00.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/530600364' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi129-82555-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:00.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:00.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: 174.5 scrub starts 2024-03-11T20:55:00.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: 174.5 scrub ok 2024-03-11T20:55:00.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: pgmap v152: 460 pgs: 64 unknown, 396 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 94 KiB/s wr, 91 op/s 2024-03-11T20:55:00.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: 174.1 scrub starts 2024-03-11T20:55:00.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: 174.1 scrub ok 2024-03-11T20:55:00.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: 174.9 scrub starts 2024-03-11T20:55:00.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: 174.9 scrub ok 2024-03-11T20:55:00.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: 174.8 scrub starts 2024-03-11T20:55:00.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: 174.8 scrub ok 2024-03-11T20:55:00.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: 174.7 deep-scrub starts 2024-03-11T20:55:00.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: 174.7 deep-scrub ok 2024-03-11T20:55:00.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: 174.2 scrub starts 2024-03-11T20:55:00.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: 174.2 scrub ok 2024-03-11T20:55:00.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: 174.0 scrub starts 2024-03-11T20:55:00.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: 174.0 scrub ok 2024-03-11T20:55:00.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: osdmap e133: 8 total, 8 up, 8 in 2024-03-11T20:55:00.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/530600364' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi129-82555-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:00.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:00 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:00.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: 174.5 scrub starts 2024-03-11T20:55:00.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: 174.5 scrub ok 2024-03-11T20:55:00.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: pgmap v152: 460 pgs: 64 unknown, 396 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 94 KiB/s wr, 91 op/s 2024-03-11T20:55:00.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: 174.1 scrub starts 2024-03-11T20:55:00.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: 174.1 scrub ok 2024-03-11T20:55:00.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: 174.9 scrub starts 2024-03-11T20:55:00.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: 174.9 scrub ok 2024-03-11T20:55:00.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: 174.8 scrub starts 2024-03-11T20:55:00.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: 174.8 scrub ok 2024-03-11T20:55:00.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: 174.7 deep-scrub starts 2024-03-11T20:55:00.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: 174.7 deep-scrub ok 2024-03-11T20:55:00.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: 174.2 scrub starts 2024-03-11T20:55:00.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: 174.2 scrub ok 2024-03-11T20:55:00.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: 174.0 scrub starts 2024-03-11T20:55:00.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: 174.0 scrub ok 2024-03-11T20:55:00.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: osdmap e133: 8 total, 8 up, 8 in 2024-03-11T20:55:00.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/530600364' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi129-82555-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:00.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:00 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:01.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:01 smithi129 ceph-mon[30184]: 174.3 scrub starts 2024-03-11T20:55:01.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:01 smithi129 ceph-mon[30184]: 174.3 scrub ok 2024-03-11T20:55:01.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/530600364' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi129-82555-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:01.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:01 smithi129 ceph-mon[30184]: osdmap e134: 8 total, 8 up, 8 in 2024-03-11T20:55:01.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2739000156' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi129-82567-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:01.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:01 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi129-82567-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:01.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:01.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:01 smithi129 ceph-mon[42066]: 174.3 scrub starts 2024-03-11T20:55:01.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:01 smithi129 ceph-mon[42066]: 174.3 scrub ok 2024-03-11T20:55:01.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/530600364' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi129-82555-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:01.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:01 smithi129 ceph-mon[42066]: osdmap e134: 8 total, 8 up, 8 in 2024-03-11T20:55:01.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2739000156' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi129-82567-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:01.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:01 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi129-82567-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:01.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:01.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:01 smithi157 ceph-mon[33359]: 174.3 scrub starts 2024-03-11T20:55:01.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:01 smithi157 ceph-mon[33359]: 174.3 scrub ok 2024-03-11T20:55:01.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/530600364' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi129-82555-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:01.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:01 smithi157 ceph-mon[33359]: osdmap e134: 8 total, 8 up, 8 in 2024-03-11T20:55:01.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2739000156' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi129-82567-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:01.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:01 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi129-82567-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:01.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:02.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:02 smithi129 ceph-mon[30184]: pgmap v155: 460 pgs: 64 unknown, 396 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:55:02.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:02 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi129-82567-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:02.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:02 smithi129 ceph-mon[30184]: osdmap e135: 8 total, 8 up, 8 in 2024-03-11T20:55:02.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:02 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:02 smithi129 ceph-mon[42066]: pgmap v155: 460 pgs: 64 unknown, 396 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:55:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:02 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi129-82567-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:02 smithi129 ceph-mon[42066]: osdmap e135: 8 total, 8 up, 8 in 2024-03-11T20:55:02.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:02 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:02 smithi157 ceph-mon[33359]: pgmap v155: 460 pgs: 64 unknown, 396 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:55:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:02 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi129-82567-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:02 smithi157 ceph-mon[33359]: osdmap e135: 8 total, 8 up, 8 in 2024-03-11T20:55:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:02 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:03.688 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:55:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:55:03.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:03 smithi129 ceph-mon[30184]: osdmap e136: 8 total, 8 up, 8 in 2024-03-11T20:55:03.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:03 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:03.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1036726490' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi129-82555-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:03.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:03.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:03 smithi129 ceph-mon[42066]: osdmap e136: 8 total, 8 up, 8 in 2024-03-11T20:55:03.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:03 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:03.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1036726490' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi129-82555-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:03.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:03.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:03 smithi157 ceph-mon[33359]: osdmap e136: 8 total, 8 up, 8 in 2024-03-11T20:55:03.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:03 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:03.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1036726490' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi129-82555-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:03.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:04.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[30184]: pgmap v158: 460 pgs: 32 creating+peering, 1 active+clean+snaptrim, 32 unknown, 395 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.0 KiB/s wr, 77 op/s 2024-03-11T20:55:04.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:04.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[30184]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:04.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1036726490' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi129-82555-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:04.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[30184]: osdmap e137: 8 total, 8 up, 8 in 2024-03-11T20:55:04.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:55:04.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:55:04.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2632567440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi129-82567-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:04.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi129-82567-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:04.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:04.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[42066]: pgmap v158: 460 pgs: 32 creating+peering, 1 active+clean+snaptrim, 32 unknown, 395 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.0 KiB/s wr, 77 op/s 2024-03-11T20:55:04.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:04.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[42066]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:04.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1036726490' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi129-82555-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:04.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[42066]: osdmap e137: 8 total, 8 up, 8 in 2024-03-11T20:55:04.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:55:04.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:55:04.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2632567440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi129-82567-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:04.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi129-82567-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:04.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:04.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:04 smithi157 ceph-mon[33359]: pgmap v158: 460 pgs: 32 creating+peering, 1 active+clean+snaptrim, 32 unknown, 395 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 4.0 KiB/s wr, 77 op/s 2024-03-11T20:55:04.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:04 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:04.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:04 smithi157 ceph-mon[33359]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:04.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1036726490' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi129-82555-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:04 smithi157 ceph-mon[33359]: osdmap e137: 8 total, 8 up, 8 in 2024-03-11T20:55:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:55:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:55:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2632567440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi129-82567-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi129-82567-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:05.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started api_snapshots: Running main() from gmock_main.cc 2024-03-11T20:55:05.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-03-11T20:55:05.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] Global test environment set-up. 2024-03-11T20:55:05.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-03-11T20:55:05.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-03-11T20:55:05.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (5028 ms) 2024-03-11T20:55:05.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-03-11T20:55:05.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2527 ms) 2024-03-11T20:55:05.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-03-11T20:55:05.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (3274 ms) 2024-03-11T20:55:05.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-03-11T20:55:05.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2618 ms) 2024-03-11T20:55:05.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (13447 ms total) 2024-03-11T20:55:05.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: 2024-03-11T20:55:05.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-03-11T20:55:05.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-03-11T20:55:05.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (5400 ms) 2024-03-11T20:55:05.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-03-11T20:55:05.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4111 ms) 2024-03-11T20:55:05.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-03-11T20:55:05.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (6858 ms) 2024-03-11T20:55:05.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (16369 ms total) 2024-03-11T20:55:05.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: 2024-03-11T20:55:05.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-03-11T20:55:05.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-03-11T20:55:05.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3040 ms) 2024-03-11T20:55:05.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-03-11T20:55:05.706 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1958 ms) 2024-03-11T20:55:05.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-03-11T20:55:05.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2003 ms) 2024-03-11T20:55:05.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-03-11T20:55:05.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2008 ms) 2024-03-11T20:55:05.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9009 ms total) 2024-03-11T20:55:05.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: 2024-03-11T20:55:05.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-03-11T20:55:05.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-03-11T20:55:05.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4047 ms) 2024-03-11T20:55:05.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-03-11T20:55:05.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4030 ms) 2024-03-11T20:55:05.707 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8077 ms total) 2024-03-11T20:55:05.708 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: 2024-03-11T20:55:05.708 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [----------] Global test environment tear-down 2024-03-11T20:55:05.708 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (71910 ms total) 2024-03-11T20:55:05.708 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-03-11T20:55:05.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]': finished 2024-03-11T20:55:05.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi129-82567-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:05.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[30184]: osdmap e138: 8 total, 8 up, 8 in 2024-03-11T20:55:05.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:55:05.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:55:05.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/412936246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82913-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:05.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:05.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[30184]: pgmap v161: 452 pgs: 32 creating+peering, 64 unknown, 356 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 73 op/s 2024-03-11T20:55:05.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]': finished 2024-03-11T20:55:05.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/412936246' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82913-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:05.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[30184]: osdmap e139: 8 total, 8 up, 8 in 2024-03-11T20:55:05.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2699942554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi129-82555-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:05.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]': finished 2024-03-11T20:55:05.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi129-82567-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:05.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[42066]: osdmap e138: 8 total, 8 up, 8 in 2024-03-11T20:55:05.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:55:05.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:55:05.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/412936246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82913-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:05.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:05.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[42066]: pgmap v161: 452 pgs: 32 creating+peering, 64 unknown, 356 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 73 op/s 2024-03-11T20:55:05.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]': finished 2024-03-11T20:55:05.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/412936246' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82913-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:05.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[42066]: osdmap e139: 8 total, 8 up, 8 in 2024-03-11T20:55:05.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2699942554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi129-82555-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:05.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:05 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]': finished 2024-03-11T20:55:05.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:05 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi129-82567-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:05.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:05 smithi157 ceph-mon[33359]: osdmap e138: 8 total, 8 up, 8 in 2024-03-11T20:55:05.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3287463555' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:55:05.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:05 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]: dispatch 2024-03-11T20:55:05.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/412936246' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82913-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:05.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:05.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:05 smithi157 ceph-mon[33359]: pgmap v161: 452 pgs: 32 creating+peering, 64 unknown, 356 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 73 op/s 2024-03-11T20:55:05.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:05 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi129-82945-15"}]': finished 2024-03-11T20:55:05.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/412936246' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82913-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:05.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:05 smithi157 ceph-mon[33359]: osdmap e139: 8 total, 8 up, 8 in 2024-03-11T20:55:05.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2699942554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi129-82555-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:06.478 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:55:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:55:06.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:06.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T20:55:06.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:06.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:06 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:06.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2699942554' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi129-82555-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:06.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:06 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:06.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:06 smithi157 ceph-mon[33359]: osdmap e140: 8 total, 8 up, 8 in 2024-03-11T20:55:06.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app2"}]: dispatch 2024-03-11T20:55:06.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:06.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/712612419' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi129-82567-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:06.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:06 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:06.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:06 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi129-82567-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:07.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:07.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T20:55:07.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:07.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:07.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2699942554' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi129-82555-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:07.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:07.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[30184]: osdmap e140: 8 total, 8 up, 8 in 2024-03-11T20:55:07.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app2"}]: dispatch 2024-03-11T20:55:07.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:07.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/712612419' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi129-82567-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:07.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:07.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi129-82567-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:07.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:07.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T20:55:07.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:07.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:07.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2699942554' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi129-82555-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:07.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:07.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[42066]: osdmap e140: 8 total, 8 up, 8 in 2024-03-11T20:55:07.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app2"}]: dispatch 2024-03-11T20:55:07.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:07.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/712612419' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi129-82567-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:07.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:07.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:06 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi129-82567-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:07 smithi129 ceph-mon[30184]: pgmap v164: 484 pgs: 37 creating+peering, 9 unknown, 438 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:55:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi129-82567-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:07 smithi129 ceph-mon[30184]: osdmap e141: 8 total, 8 up, 8 in 2024-03-11T20:55:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-11T20:55:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T20:55:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T20:55:08.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:08.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:07 smithi129 ceph-mon[42066]: pgmap v164: 484 pgs: 37 creating+peering, 9 unknown, 438 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:55:08.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:08.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi129-82567-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:08.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:07 smithi129 ceph-mon[42066]: osdmap e141: 8 total, 8 up, 8 in 2024-03-11T20:55:08.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-11T20:55:08.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T20:55:08.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T20:55:08.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:08.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:07 smithi157 ceph-mon[33359]: pgmap v164: 484 pgs: 37 creating+peering, 9 unknown, 438 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:55:08.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi129-82724-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:08.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi129-82567-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:08.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:07 smithi157 ceph-mon[33359]: osdmap e141: 8 total, 8 up, 8 in 2024-03-11T20:55:08.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-11T20:55:08.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T20:55:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T20:55:09.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:09.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-11T20:55:09.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:08 smithi129 ceph-mon[30184]: osdmap e142: 8 total, 8 up, 8 in 2024-03-11T20:55:09.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1567411669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi129-82555-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:09.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T20:55:09.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi129-82555-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:09.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T20:55:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-11T20:55:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:08 smithi129 ceph-mon[42066]: osdmap e142: 8 total, 8 up, 8 in 2024-03-11T20:55:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1567411669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi129-82555-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T20:55:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi129-82555-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T20:55:09.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:09.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-11T20:55:09.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:08 smithi157 ceph-mon[33359]: osdmap e142: 8 total, 8 up, 8 in 2024-03-11T20:55:09.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1567411669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi129-82555-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:09.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T20:55:09.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi129-82555-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:09.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T20:55:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: pgmap v167: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T20:55:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:10.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:10.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi129-82555-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:10.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-11T20:55:10.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:10.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: osdmap e143: 8 total, 8 up, 8 in 2024-03-11T20:55:10.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi129-82913-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T20:55:10.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T20:55:10.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi129-82913-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2400407999' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi129-82567-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:10.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:10.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: pgmap v167: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T20:55:10.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:10.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:10.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:10.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi129-82555-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:10.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-11T20:55:10.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:10.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: osdmap e143: 8 total, 8 up, 8 in 2024-03-11T20:55:10.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi129-82913-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T20:55:10.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T20:55:10.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi129-82913-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2400407999' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi129-82567-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:10.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:10.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: pgmap v167: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T20:55:10.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:10.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:10.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:10.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi129-82555-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:10.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-11T20:55:10.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:10.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: osdmap e143: 8 total, 8 up, 8 in 2024-03-11T20:55:10.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi129-82913-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/834917808' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T20:55:10.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T20:55:10.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi129-82913-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:10.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2400407999' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi129-82567-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:11.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:11.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1"}]': finished 2024-03-11T20:55:11.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:11.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1"}]': finished 2024-03-11T20:55:11.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:11.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi129-82724-1","app":"app1","key":"key1"}]': finished 2024-03-11T20:55:11.718 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-03-11T20:55:11.718 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-03-11T20:55:11.718 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-03-11T20:55:11.718 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-03-11T20:55:11.718 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-03-11T20:55:11.718 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-11T20:55:11.718 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2024-03-11T20:55:11.719 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-03-11T20:55:11.719 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: seed 82724 2024-03-11T20:55:11.719 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-03-11T20:55:11.719 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2024-03-11T20:55:11.719 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-03-11T20:55:11.719 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (385 ms) 2024-03-11T20:55:11.719 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-03-11T20:55:11.719 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (13 ms) 2024-03-11T20:55:11.719 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-03-11T20:55:11.719 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (10 ms) 2024-03-11T20:55:11.719 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-03-11T20:55:11.719 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (11 ms) 2024-03-11T20:55:11.719 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-03-11T20:55:11.720 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (3 ms) 2024-03-11T20:55:11.720 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-03-11T20:55:11.720 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (3 ms) 2024-03-11T20:55:11.720 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-03-11T20:55:11.720 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (8 ms) 2024-03-11T20:55:11.720 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-03-11T20:55:11.720 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (3 ms) 2024-03-11T20:55:11.720 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-03-11T20:55:11.720 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (12 ms) 2024-03-11T20:55:11.720 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-03-11T20:55:11.720 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (2 ms) 2024-03-11T20:55:11.720 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-03-11T20:55:11.720 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (5 ms) 2024-03-11T20:55:11.720 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-03-11T20:55:11.720 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (11 ms) 2024-03-11T20:55:11.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-03-11T20:55:11.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: osd_max_attr_size = 0 2024-03-11T20:55:11.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-03-11T20:55:11.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1539 ms) 2024-03-11T20:55:11.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-03-11T20:55:11.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (536 ms) 2024-03-11T20:55:11.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-03-11T20:55:11.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-03-11T20:55:11.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: done waiting, doing copies 2024-03-11T20:55:11.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: waiting for final deep scrubs... 2024-03-11T20:55:11.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: done waiting 2024-03-11T20:55:11.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60640 ms) 2024-03-11T20:55:11.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-03-11T20:55:11.721 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (3 ms) 2024-03-11T20:55:11.722 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-03-11T20:55:11.722 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (3 ms) 2024-03-11T20:55:11.722 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-03-11T20:55:11.722 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4750 ms) 2024-03-11T20:55:11.722 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-03-11T20:55:11.722 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-03-11T20:55:11.722 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-03-11T20:55:11.722 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-03-11T20:55:11.722 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-03-11T20:55:11.722 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-03-11T20:55:12.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2400407999' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi129-82567-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:12.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:12.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[30184]: pgmap v170: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:55:12.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[30184]: osdmap e144: 8 total, 8 up, 8 in 2024-03-11T20:55:12.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi129-82913-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]': finished 2024-03-11T20:55:12.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[30184]: osdmap e145: 8 total, 8 up, 8 in 2024-03-11T20:55:12.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2447554207' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi129-82555-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:12.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi129-82555-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:12.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:12.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:12.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:12.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:12.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:12.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82724-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:12.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82724-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:12.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2400407999' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi129-82567-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:12.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:12.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[42066]: pgmap v170: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:55:12.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[42066]: osdmap e144: 8 total, 8 up, 8 in 2024-03-11T20:55:12.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi129-82913-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]': finished 2024-03-11T20:55:12.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[42066]: osdmap e145: 8 total, 8 up, 8 in 2024-03-11T20:55:12.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2447554207' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi129-82555-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:12.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi129-82555-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:12.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:12.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:12.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:12.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:12.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:12.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82724-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:12.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82724-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:12.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2400407999' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi129-82567-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:12.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:12.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:12 smithi157 ceph-mon[33359]: pgmap v170: 420 pgs: 32 unknown, 388 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:55:12.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:12 smithi157 ceph-mon[33359]: osdmap e144: 8 total, 8 up, 8 in 2024-03-11T20:55:12.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi129-82913-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]': finished 2024-03-11T20:55:12.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:12 smithi157 ceph-mon[33359]: osdmap e145: 8 total, 8 up, 8 in 2024-03-11T20:55:12.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2447554207' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi129-82555-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:12.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi129-82555-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:12.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:12.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:12.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:12.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:12.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:12.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82724-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:12.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82724-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:13.717 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:55:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:55:13.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi129-82555-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:13.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82724-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:13.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:13 smithi129 ceph-mon[30184]: osdmap e146: 8 total, 8 up, 8 in 2024-03-11T20:55:13.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82724-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:13.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82724-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:13.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/226510143' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi129-82567-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:13.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi129-82555-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:13.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82724-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:13 smithi129 ceph-mon[42066]: osdmap e146: 8 total, 8 up, 8 in 2024-03-11T20:55:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82724-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82724-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/226510143' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi129-82567-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi129-82555-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82724-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:13 smithi157 ceph-mon[33359]: osdmap e146: 8 total, 8 up, 8 in 2024-03-11T20:55:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82724-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82724-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/226510143' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi129-82567-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:13.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:15.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:14 smithi129 ceph-mon[42066]: pgmap v173: 428 pgs: 32 unknown, 40 creating+peering, 356 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:55:15.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:14 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:15.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/226510143' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi129-82567-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:14 smithi129 ceph-mon[42066]: osdmap e147: 8 total, 8 up, 8 in 2024-03-11T20:55:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:14 smithi129 ceph-mon[42066]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82724-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82724-24"}]': finished 2024-03-11T20:55:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:14 smithi129 ceph-mon[42066]: osdmap e148: 8 total, 8 up, 8 in 2024-03-11T20:55:15.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:14 smithi129 ceph-mon[30184]: pgmap v173: 428 pgs: 32 unknown, 40 creating+peering, 356 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:55:15.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:14 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:15.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/226510143' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi129-82567-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:15.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:15.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:14 smithi129 ceph-mon[30184]: osdmap e147: 8 total, 8 up, 8 in 2024-03-11T20:55:15.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:14 smithi129 ceph-mon[30184]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:15.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82724-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82724-24"}]': finished 2024-03-11T20:55:15.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:14 smithi129 ceph-mon[30184]: osdmap e148: 8 total, 8 up, 8 in 2024-03-11T20:55:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:14 smithi157 ceph-mon[33359]: pgmap v173: 428 pgs: 32 unknown, 40 creating+peering, 356 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:55:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:14 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/226510143' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi129-82567-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:15.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:14 smithi157 ceph-mon[33359]: osdmap e147: 8 total, 8 up, 8 in 2024-03-11T20:55:15.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:14 smithi157 ceph-mon[33359]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:15.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82724-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82724-24"}]': finished 2024-03-11T20:55:15.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:14 smithi157 ceph-mon[33359]: osdmap e148: 8 total, 8 up, 8 in 2024-03-11T20:55:16.229 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:55:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:55:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:55:16.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2621424803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi129-82555-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:16.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[42066]: pgmap v176: 404 pgs: 40 unknown, 8 creating+peering, 356 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:55:16.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi129-82555-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi129-82555-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:16.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[42066]: osdmap e149: 8 total, 8 up, 8 in 2024-03-11T20:55:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1584344739' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi129-82567-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi129-82567-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:16.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2621424803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi129-82555-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:16.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[30184]: pgmap v176: 404 pgs: 40 unknown, 8 creating+peering, 356 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:55:16.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi129-82555-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi129-82555-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:16.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[30184]: osdmap e149: 8 total, 8 up, 8 in 2024-03-11T20:55:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1584344739' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi129-82567-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi129-82567-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:16.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2621424803' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi129-82555-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:15 smithi157 ceph-mon[33359]: pgmap v176: 404 pgs: 40 unknown, 8 creating+peering, 356 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-11T20:55:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi129-82555-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi129-82555-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:15 smithi157 ceph-mon[33359]: osdmap e149: 8 total, 8 up, 8 in 2024-03-11T20:55:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1584344739' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi129-82567-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi129-82567-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:16.662 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscP api_snapshots_pp: Running main() from gmock_main.cc 2024-03-11T20:55:16.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-03-11T20:55:16.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-03-11T20:55:16.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-03-11T20:55:16.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: seed 82913 2024-03-11T20:55:16.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-03-11T20:55:16.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (5028 ms) 2024-03-11T20:55:16.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-03-11T20:55:16.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2528 ms) 2024-03-11T20:55:16.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-03-11T20:55:16.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (3275 ms) 2024-03-11T20:55:16.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-03-11T20:55:16.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2616 ms) 2024-03-11T20:55:16.663 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-03-11T20:55:16.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (5125 ms) 2024-03-11T20:55:16.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (18572 ms total) 2024-03-11T20:55:16.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: 2024-03-11T20:55:16.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-03-11T20:55:16.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-03-11T20:55:16.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4198 ms) 2024-03-11T20:55:16.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-03-11T20:55:16.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4814 ms) 2024-03-11T20:55:16.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-03-11T20:55:16.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (5990 ms) 2024-03-11T20:55:16.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-03-11T20:55:16.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4013 ms) 2024-03-11T20:55:16.664 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-03-11T20:55:16.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2046 ms) 2024-03-11T20:55:16.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-11T20:55:16.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1979-g2b0101eb/rpm/el9/BUILD/ceph-19.0.0-1979-g2b0101eb/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-03-11T20:55:16.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: 2024-03-11T20:55:16.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-03-11T20:55:16.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-03-11T20:55:16.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi129-82913-7 2024-03-11T20:55:16.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: waiting for snaps to purge 2024-03-11T20:55:16.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17952 ms) 2024-03-11T20:55:16.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (39013 ms total) 2024-03-11T20:55:16.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: 2024-03-11T20:55:16.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-03-11T20:55:16.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-03-11T20:55:16.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (9 ms) 2024-03-11T20:55:16.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-03-11T20:55:16.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5039 ms) 2024-03-11T20:55:16.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5048 ms total) 2024-03-11T20:55:16.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: 2024-03-11T20:55:16.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-03-11T20:55:16.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-03-11T20:55:16.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2940 ms) 2024-03-11T20:55:16.666 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-03-11T20:55:17.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi129-82567-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:17.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:17.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:17 smithi129 ceph-mon[30184]: osdmap e150: 8 total, 8 up, 8 in 2024-03-11T20:55:17.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:17.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:17.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:17.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi129-82567-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:17.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:17.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:17 smithi129 ceph-mon[42066]: osdmap e150: 8 total, 8 up, 8 in 2024-03-11T20:55:17.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:17.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:17.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:17.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi129-82567-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:17.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:17.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:17 smithi157 ceph-mon[33359]: osdmap e150: 8 total, 8 up, 8 in 2024-03-11T20:55:17.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:17.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:17.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:18 smithi129 ceph-mon[30184]: pgmap v179: 436 pgs: 436 active+clean; 592 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T20:55:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:18 smithi129 ceph-mon[30184]: osdmap e151: 8 total, 8 up, 8 in 2024-03-11T20:55:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3215240502' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi129-82555-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T20:55:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:18 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T20:55:18.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:18 smithi129 ceph-mon[42066]: pgmap v179: 436 pgs: 436 active+clean; 592 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T20:55:18.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:18.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:18 smithi129 ceph-mon[42066]: osdmap e151: 8 total, 8 up, 8 in 2024-03-11T20:55:18.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3215240502' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi129-82555-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:18.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:18.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T20:55:18.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:18 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:18.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T20:55:18.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:18 smithi157 ceph-mon[33359]: pgmap v179: 436 pgs: 436 active+clean; 592 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T20:55:18.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82724-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:18.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:18 smithi157 ceph-mon[33359]: osdmap e151: 8 total, 8 up, 8 in 2024-03-11T20:55:18.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3215240502' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi129-82555-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:18.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:18.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T20:55:18.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:18 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:18.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T20:55:19.690 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3215240502' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi129-82555-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:19.690 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]': finished 2024-03-11T20:55:19.690 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:19 smithi157 ceph-mon[33359]: osdmap e152: 8 total, 8 up, 8 in 2024-03-11T20:55:19.690 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3500919930' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi129-82567-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:19.690 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:19.690 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:19.691 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T20:55:19.691 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T20:55:19.691 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3500919930' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi129-82567-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:19.691 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]': finished 2024-03-11T20:55:19.691 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:19 smithi157 ceph-mon[33359]: osdmap e153: 8 total, 8 up, 8 in 2024-03-11T20:55:19.691 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:19.691 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:19.691 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:19.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3215240502' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi129-82555-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:19.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]': finished 2024-03-11T20:55:19.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[30184]: osdmap e152: 8 total, 8 up, 8 in 2024-03-11T20:55:19.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3500919930' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi129-82567-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:19.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:19.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:19.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T20:55:19.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T20:55:19.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3500919930' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi129-82567-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:19.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]': finished 2024-03-11T20:55:19.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[30184]: osdmap e153: 8 total, 8 up, 8 in 2024-03-11T20:55:19.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:19.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:19.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:19.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3215240502' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi129-82555-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:19.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]': finished 2024-03-11T20:55:19.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[42066]: osdmap e152: 8 total, 8 up, 8 in 2024-03-11T20:55:19.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3500919930' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi129-82567-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:19.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:19.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:19.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T20:55:19.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T20:55:19.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3500919930' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi129-82567-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:19.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]': finished 2024-03-11T20:55:19.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[42066]: osdmap e153: 8 total, 8 up, 8 in 2024-03-11T20:55:19.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:19.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:19.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:20.665 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnaps snapshots: Running main() from gmock_main.cc 2024-03-11T20:55:20.665 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-03-11T20:55:20.665 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [----------] Global test environment set-up. 2024-03-11T20:55:20.665 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-03-11T20:55:20.665 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-03-11T20:55:20.665 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (13933 ms) 2024-03-11T20:55:20.665 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-03-11T20:55:20.665 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (6105 ms) 2024-03-11T20:55:20.665 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-03-11T20:55:20.665 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (4141 ms) 2024-03-11T20:55:20.666 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-03-11T20:55:20.666 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5344 ms) 2024-03-11T20:55:20.666 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-03-11T20:55:20.666 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (6990 ms) 2024-03-11T20:55:20.666 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (36514 ms total) 2024-03-11T20:55:20.666 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: 2024-03-11T20:55:20.666 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-03-11T20:55:20.666 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-03-11T20:55:20.666 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5005 ms) 2024-03-11T20:55:20.666 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-03-11T20:55:20.666 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (7020 ms) 2024-03-11T20:55:20.666 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-03-11T20:55:20.666 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (7995 ms) 2024-03-11T20:55:20.666 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-03-11T20:55:20.667 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6011 ms) 2024-03-11T20:55:20.667 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-03-11T20:55:20.667 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4011 ms) 2024-03-11T20:55:20.667 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-03-11T20:55:20.667 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi129-83909-11. 2024-03-11T20:55:20.667 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: Waiting for snaps to purge. 2024-03-11T20:55:20.667 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (19986 ms) 2024-03-11T20:55:20.667 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (50028 ms total) 2024-03-11T20:55:20.667 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: 2024-03-11T20:55:20.667 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [----------] Global test environment tear-down 2024-03-11T20:55:20.667 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (86542 ms total) 2024-03-11T20:55:20.667 INFO:tasks.workunit.client.0.smithi129.stdout: snapshots: [ PASSED ] 11 tests. 2024-03-11T20:55:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:20 smithi129 ceph-mon[30184]: pgmap v182: 436 pgs: 64 unknown, 372 active+clean; 592 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-11T20:55:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82724-24"}]': finished 2024-03-11T20:55:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:20 smithi129 ceph-mon[30184]: osdmap e154: 8 total, 8 up, 8 in 2024-03-11T20:55:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:20.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:20.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3243050765' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi129-82555-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:20.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi129-82555-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:20.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:20 smithi129 ceph-mon[42066]: pgmap v182: 436 pgs: 64 unknown, 372 active+clean; 592 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-11T20:55:20.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:20.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82724-24"}]': finished 2024-03-11T20:55:20.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:20 smithi129 ceph-mon[42066]: osdmap e154: 8 total, 8 up, 8 in 2024-03-11T20:55:20.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:20.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:20.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3243050765' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi129-82555-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:20.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi129-82555-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:20.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:20 smithi157 ceph-mon[33359]: pgmap v182: 436 pgs: 64 unknown, 372 active+clean; 592 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-11T20:55:20.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:20.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82724-24"}]': finished 2024-03-11T20:55:20.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:20 smithi157 ceph-mon[33359]: osdmap e154: 8 total, 8 up, 8 in 2024-03-11T20:55:20.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3449127105' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:20.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82724-24"}]: dispatch 2024-03-11T20:55:20.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3243050765' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi129-82555-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:20.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi129-82555-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:21.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:21.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82724-24"}]': finished 2024-03-11T20:55:21.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi129-82555-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:21.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:21 smithi129 ceph-mon[30184]: osdmap e155: 8 total, 8 up, 8 in 2024-03-11T20:55:21.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1385309031' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi129-82567-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:21.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi129-82567-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:21.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:21.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:21.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:21.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82724-24"}]': finished 2024-03-11T20:55:21.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi129-82555-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:21.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:21 smithi129 ceph-mon[42066]: osdmap e155: 8 total, 8 up, 8 in 2024-03-11T20:55:21.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1385309031' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi129-82567-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:21.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi129-82567-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:21.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:21.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:21.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:21.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82724-24"}]': finished 2024-03-11T20:55:21.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi129-82555-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:21.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:21 smithi157 ceph-mon[33359]: osdmap e155: 8 total, 8 up, 8 in 2024-03-11T20:55:21.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1385309031' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi129-82567-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:21.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi129-82567-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:21.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:21.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:22.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[30184]: pgmap v185: 364 pgs: 32 unknown, 332 active+clean; 592 KiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:55:22.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[30184]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:22.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:22.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi129-82567-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:22.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]': finished 2024-03-11T20:55:22.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[30184]: osdmap e156: 8 total, 8 up, 8 in 2024-03-11T20:55:22.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:22.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/32305617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi129-82724-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:22.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[42066]: pgmap v185: 364 pgs: 32 unknown, 332 active+clean; 592 KiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:55:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[42066]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi129-82567-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:22.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]': finished 2024-03-11T20:55:22.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[42066]: osdmap e156: 8 total, 8 up, 8 in 2024-03-11T20:55:22.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:22.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/32305617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi129-82724-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:22.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:22.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:22 smithi157 ceph-mon[33359]: pgmap v185: 364 pgs: 32 unknown, 332 active+clean; 592 KiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:55:22.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:22 smithi157 ceph-mon[33359]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:22.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:22.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi129-82567-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:22.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi129-82913-16"}]': finished 2024-03-11T20:55:22.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:22 smithi157 ceph-mon[33359]: osdmap e156: 8 total, 8 up, 8 in 2024-03-11T20:55:22.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4037957819' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:22.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/32305617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi129-82724-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:22.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi129-82913-16"}]: dispatch 2024-03-11T20:55:23.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:55:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:55:23.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/32305617' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi129-82724-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:23 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi129-82913-16"}]': finished 2024-03-11T20:55:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:23 smithi129 ceph-mon[30184]: osdmap e157: 8 total, 8 up, 8 in 2024-03-11T20:55:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/679764261' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi129-82555-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:23 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi129-82555-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:23.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:23.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/32305617' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi129-82724-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:23.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:23 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi129-82913-16"}]': finished 2024-03-11T20:55:23.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:23 smithi129 ceph-mon[42066]: osdmap e157: 8 total, 8 up, 8 in 2024-03-11T20:55:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/679764261' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi129-82555-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:23 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi129-82555-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:23.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:23.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/32305617' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi129-82724-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:23.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:23 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi129-82913-16"}]': finished 2024-03-11T20:55:23.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:23 smithi157 ceph-mon[33359]: osdmap e157: 8 total, 8 up, 8 in 2024-03-11T20:55:23.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/679764261' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi129-82555-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:23.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:23 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi129-82555-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[30184]: pgmap v188: 388 pgs: 32 creating+peering, 32 unknown, 324 active+clean; 592 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-03-11T20:55:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]: dispatch 2024-03-11T20:55:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]: dispatch 2024-03-11T20:55:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi129-82555-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[30184]: osdmap e158: 8 total, 8 up, 8 in 2024-03-11T20:55:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]: dispatch 2024-03-11T20:55:25.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2905303280' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi129-82567-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:25.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[42066]: pgmap v188: 388 pgs: 32 creating+peering, 32 unknown, 324 active+clean; 592 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-03-11T20:55:25.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:25.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]: dispatch 2024-03-11T20:55:25.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]: dispatch 2024-03-11T20:55:25.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:25.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:25.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi129-82555-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:25.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:25.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[42066]: osdmap e158: 8 total, 8 up, 8 in 2024-03-11T20:55:25.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]: dispatch 2024-03-11T20:55:25.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2905303280' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi129-82567-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:24 smithi157 ceph-mon[33359]: pgmap v188: 388 pgs: 32 creating+peering, 32 unknown, 324 active+clean; 592 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 8 op/s 2024-03-11T20:55:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:24 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]: dispatch 2024-03-11T20:55:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]: dispatch 2024-03-11T20:55:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:25.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi129-82555-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:25.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:25.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:24 smithi157 ceph-mon[33359]: osdmap e158: 8 total, 8 up, 8 in 2024-03-11T20:55:25.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]: dispatch 2024-03-11T20:55:25.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2905303280' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi129-82567-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:26.228 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:55:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:55:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:55:26.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:26.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[30184]: pgmap v191: 388 pgs: 64 unknown, 324 active+clean; 592 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-03-11T20:55:26.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2905303280' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi129-82567-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:26.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:26.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[30184]: osdmap e159: 8 total, 8 up, 8 in 2024-03-11T20:55:26.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1377641359' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi129-82724-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:26.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi129-82724-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:26.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:26.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:26.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:26.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:26.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi129-82555-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:26.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi129-82555-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[42066]: pgmap v191: 388 pgs: 64 unknown, 324 active+clean; 592 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-03-11T20:55:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2905303280' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi129-82567-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[42066]: osdmap e159: 8 total, 8 up, 8 in 2024-03-11T20:55:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1377641359' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi129-82724-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi129-82724-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:26.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:26.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:26.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi129-82555-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:26.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi129-82555-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:25 smithi157 ceph-mon[33359]: pgmap v191: 388 pgs: 64 unknown, 324 active+clean; 592 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-03-11T20:55:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2905303280' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi129-82567-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:25 smithi157 ceph-mon[33359]: osdmap e159: 8 total, 8 up, 8 in 2024-03-11T20:55:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1377641359' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi129-82724-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi129-82724-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:26.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:26.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:26.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi129-82555-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:26.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi129-82555-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:27.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]': finished 2024-03-11T20:55:27.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:27 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi129-82724-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:27.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:27 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi129-82555-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:27.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:27 smithi129 ceph-mon[30184]: osdmap e160: 8 total, 8 up, 8 in 2024-03-11T20:55:27.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi129-82555-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:27.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:27 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi129-82555-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:27.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:27.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]': finished 2024-03-11T20:55:27.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:27 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi129-82724-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:27.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:27 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi129-82555-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:27.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:27 smithi129 ceph-mon[42066]: osdmap e160: 8 total, 8 up, 8 in 2024-03-11T20:55:27.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi129-82555-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:27.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:27 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi129-82555-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:27.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]': finished 2024-03-11T20:55:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:27 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi129-82724-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:27 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi129-82555-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:27 smithi157 ceph-mon[33359]: osdmap e160: 8 total, 8 up, 8 in 2024-03-11T20:55:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi129-82555-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:27 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi129-82555-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:28 smithi129 ceph-mon[30184]: pgmap v194: 364 pgs: 8 creating+peering, 356 active+clean; 600 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.0 KiB/s wr, 2 op/s 2024-03-11T20:55:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:28 smithi129 ceph-mon[30184]: osdmap e161: 8 total, 8 up, 8 in 2024-03-11T20:55:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1486955790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi129-82567-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi129-82567-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:28.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:28 smithi129 ceph-mon[42066]: pgmap v194: 364 pgs: 8 creating+peering, 356 active+clean; 600 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.0 KiB/s wr, 2 op/s 2024-03-11T20:55:28.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:28.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:28 smithi129 ceph-mon[42066]: osdmap e161: 8 total, 8 up, 8 in 2024-03-11T20:55:28.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1486955790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi129-82567-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi129-82567-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:28 smithi157 ceph-mon[33359]: pgmap v194: 364 pgs: 8 creating+peering, 356 active+clean; 600 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.0 KiB/s wr, 2 op/s 2024-03-11T20:55:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:28 smithi157 ceph-mon[33359]: osdmap e161: 8 total, 8 up, 8 in 2024-03-11T20:55:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1486955790' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi129-82567-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi129-82567-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:29.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi129-82555-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi129-82555-27"}]': finished 2024-03-11T20:55:29.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi129-82567-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:29.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:29 smithi129 ceph-mon[30184]: osdmap e162: 8 total, 8 up, 8 in 2024-03-11T20:55:29.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1513566859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi129-82724-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:29.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi129-82724-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:29.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:29.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:29 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi129-82555-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi129-82555-27"}]': finished 2024-03-11T20:55:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi129-82567-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:29 smithi129 ceph-mon[42066]: osdmap e162: 8 total, 8 up, 8 in 2024-03-11T20:55:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1513566859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi129-82724-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:29.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi129-82724-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:29.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:29.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:29 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:29.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi129-82555-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi129-82555-27"}]': finished 2024-03-11T20:55:29.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi129-82567-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:29.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:29 smithi157 ceph-mon[33359]: osdmap e162: 8 total, 8 up, 8 in 2024-03-11T20:55:29.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1513566859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi129-82724-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:29.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi129-82724-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:29.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:29.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:29 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:30.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:30 smithi129 ceph-mon[30184]: pgmap v197: 404 pgs: 72 unknown, 8 creating+peering, 324 active+clean; 592 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:55:30.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:30 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi129-82724-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:30.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:30 smithi129 ceph-mon[30184]: osdmap e163: 8 total, 8 up, 8 in 2024-03-11T20:55:30.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:30.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:30 smithi129 ceph-mon[30184]: osdmap e164: 8 total, 8 up, 8 in 2024-03-11T20:55:30.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1758390061' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi129-82567-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:30.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:30 smithi129 ceph-mon[42066]: pgmap v197: 404 pgs: 72 unknown, 8 creating+peering, 324 active+clean; 592 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:55:30.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:30 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi129-82724-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:30.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:30 smithi129 ceph-mon[42066]: osdmap e163: 8 total, 8 up, 8 in 2024-03-11T20:55:30.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:30.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:30 smithi129 ceph-mon[42066]: osdmap e164: 8 total, 8 up, 8 in 2024-03-11T20:55:30.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1758390061' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi129-82567-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:30 smithi157 ceph-mon[33359]: pgmap v197: 404 pgs: 72 unknown, 8 creating+peering, 324 active+clean; 592 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:55:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:30 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi129-82724-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:30 smithi157 ceph-mon[33359]: osdmap e163: 8 total, 8 up, 8 in 2024-03-11T20:55:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:30 smithi157 ceph-mon[33359]: osdmap e164: 8 total, 8 up, 8 in 2024-03-11T20:55:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1758390061' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi129-82567-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:31.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:31.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:31.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:31.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:31.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:31.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1758390061' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi129-82567-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:31.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi129-82555-27"}]': finished 2024-03-11T20:55:31.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:31.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: osdmap e165: 8 total, 8 up, 8 in 2024-03-11T20:55:31.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi129-82724-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:31.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi129-82724-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:31.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:31.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:31.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:31.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:31.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:31.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1758390061' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi129-82567-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:31.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi129-82555-27"}]': finished 2024-03-11T20:55:31.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:31.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: osdmap e165: 8 total, 8 up, 8 in 2024-03-11T20:55:31.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi129-82724-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:31.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi129-82724-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1758390061' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi129-82567-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi129-82555-27"}]': finished 2024-03-11T20:55:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: osdmap e165: 8 total, 8 up, 8 in 2024-03-11T20:55:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi129-82724-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2308382984' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi129-82724-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi129-82555-27"}]: dispatch 2024-03-11T20:55:32.701 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: Running main() from gmock_main.cc 2024-03-11T20:55:32.701 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-03-11T20:55:32.701 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] Global test environment set-up. 2024-03-11T20:55:32.702 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-03-11T20:55:32.702 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-03-11T20:55:32.702 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.TooBig (10411 ms) 2024-03-11T20:55:32.702 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-03-11T20:55:32.702 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (6255 ms) 2024-03-11T20:55:32.702 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-03-11T20:55:32.702 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (5888 ms) 2024-03-11T20:55:32.702 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-03-11T20:55:32.702 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3117 ms) 2024-03-11T20:55:32.702 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-03-11T20:55:32.702 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3314 ms) 2024-03-11T20:55:32.702 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-03-11T20:55:32.702 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3114 ms) 2024-03-11T20:55:32.703 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-03-11T20:55:32.703 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3006 ms) 2024-03-11T20:55:32.703 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-03-11T20:55:32.703 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (2877 ms) 2024-03-11T20:55:32.703 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-03-11T20:55:32.703 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3003 ms) 2024-03-11T20:55:32.703 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-03-11T20:55:32.703 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (2994 ms) 2024-03-11T20:55:32.703 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-03-11T20:55:32.703 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3014 ms) 2024-03-11T20:55:32.703 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-03-11T20:55:32.703 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3015 ms) 2024-03-11T20:55:32.703 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-03-11T20:55:32.703 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2987 ms) 2024-03-11T20:55:32.703 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-03-11T20:55:32.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3001 ms) 2024-03-11T20:55:32.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-03-11T20:55:32.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.Flush (2999 ms) 2024-03-11T20:55:32.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-03-11T20:55:32.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3001 ms) 2024-03-11T20:55:32.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-03-11T20:55:32.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3005 ms) 2024-03-11T20:55:32.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-03-11T20:55:32.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3021 ms) 2024-03-11T20:55:32.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-03-11T20:55:32.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3025 ms) 2024-03-11T20:55:32.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-03-11T20:55:32.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3000 ms) 2024-03-11T20:55:32.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-03-11T20:55:32.704 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3251 ms) 2024-03-11T20:55:32.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-03-11T20:55:32.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3010 ms) 2024-03-11T20:55:32.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-03-11T20:55:32.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2697 ms) 2024-03-11T20:55:32.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-03-11T20:55:32.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2991 ms) 2024-03-11T20:55:32.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-03-11T20:55:32.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3014 ms) 2024-03-11T20:55:32.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-03-11T20:55:32.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3254 ms) 2024-03-11T20:55:32.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] 26 tests from LibRadosAio (92264 ms total) 2024-03-11T20:55:32.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: 2024-03-11T20:55:32.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-03-11T20:55:32.705 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-03-11T20:55:33.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:32 smithi129 ceph-mon[42066]: pgmap v200: 364 pgs: 32 unknown, 8 creating+peering, 324 active+clean; 592 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:55:33.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:32 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:33.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:32 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:33.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:32 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:33.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:32 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi129-82555-27"}]': finished 2024-03-11T20:55:33.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:32 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:55:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:32 smithi129 ceph-mon[30184]: pgmap v200: 364 pgs: 32 unknown, 8 creating+peering, 324 active+clean; 592 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:55:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:32 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:32 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:32 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:32 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi129-82555-27"}]': finished 2024-03-11T20:55:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:32 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:55:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:32 smithi157 ceph-mon[33359]: pgmap v200: 364 pgs: 32 unknown, 8 creating+peering, 324 active+clean; 592 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:55:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:32 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:32 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:32 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:32 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi129-82555-27"}]': finished 2024-03-11T20:55:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:32 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:55:33.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:55:34.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13"}]: dispatch 2024-03-11T20:55:34.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:34.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: pgmap v203: 332 pgs: 13 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 307 active+clean; 611 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 3 op/s 2024-03-11T20:55:34.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: osdmap e166: 8 total, 8 up, 8 in 2024-03-11T20:55:34.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13"}]: dispatch 2024-03-11T20:55:34.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-11T20:55:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi129-82555-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi129-82555-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:55:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi129-82724-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi129-82724-36"}]': finished 2024-03-11T20:55:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13"}]': finished 2024-03-11T20:55:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-11T20:55:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi129-82555-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: osdmap e167: 8 total, 8 up, 8 in 2024-03-11T20:55:34.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13"}]: dispatch 2024-03-11T20:55:34.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:34.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: pgmap v203: 332 pgs: 13 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 307 active+clean; 611 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 3 op/s 2024-03-11T20:55:34.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: osdmap e166: 8 total, 8 up, 8 in 2024-03-11T20:55:34.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13"}]: dispatch 2024-03-11T20:55:34.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-11T20:55:34.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi129-82555-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:34.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi129-82555-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:34.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:34.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:34.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:55:34.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi129-82724-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi129-82724-36"}]': finished 2024-03-11T20:55:34.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13"}]': finished 2024-03-11T20:55:34.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-11T20:55:34.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi129-82555-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:34.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: osdmap e167: 8 total, 8 up, 8 in 2024-03-11T20:55:34.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi129-82555-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi129-82555-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1278633469' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi129-82567-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:34.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:34.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi129-82555-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi129-82555-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1278633469' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi129-82567-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:34.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13"}]: dispatch 2024-03-11T20:55:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: pgmap v203: 332 pgs: 13 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 307 active+clean; 611 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 3 op/s 2024-03-11T20:55:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: osdmap e166: 8 total, 8 up, 8 in 2024-03-11T20:55:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13"}]: dispatch 2024-03-11T20:55:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-11T20:55:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi129-82555-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi129-82555-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:55:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi129-82724-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi129-82724-36"}]': finished 2024-03-11T20:55:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-13"}]': finished 2024-03-11T20:55:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-11T20:55:34.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi129-82555-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:34.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: osdmap e167: 8 total, 8 up, 8 in 2024-03-11T20:55:34.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi129-82555-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi129-82555-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:34.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1278633469' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi129-82567-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:34.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:35.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1278633469' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi129-82567-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:35.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:35.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:35 smithi129 ceph-mon[30184]: osdmap e168: 8 total, 8 up, 8 in 2024-03-11T20:55:35.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1278633469' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi129-82567-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:35.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:35.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:35 smithi129 ceph-mon[42066]: osdmap e168: 8 total, 8 up, 8 in 2024-03-11T20:55:35.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1278633469' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi129-82567-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:35.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:35.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:35 smithi157 ceph-mon[33359]: osdmap e168: 8 total, 8 up, 8 in 2024-03-11T20:55:36.478 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:55:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:55:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[30184]: pgmap v206: 340 pgs: 9 active+clean+snaptrim, 40 unknown, 6 active+clean+snaptrim_wait, 285 active+clean; 610 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s wr, 1 op/s 2024-03-11T20:55:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[30184]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi129-82555-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi129-82555-28"}]': finished 2024-03-11T20:55:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[30184]: osdmap e169: 8 total, 8 up, 8 in 2024-03-11T20:55:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:36.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[42066]: pgmap v206: 340 pgs: 9 active+clean+snaptrim, 40 unknown, 6 active+clean+snaptrim_wait, 285 active+clean; 610 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s wr, 1 op/s 2024-03-11T20:55:36.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[42066]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:36.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi129-82555-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi129-82555-28"}]': finished 2024-03-11T20:55:36.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[42066]: osdmap e169: 8 total, 8 up, 8 in 2024-03-11T20:55:36.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:36.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:36.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:36.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:36.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:36 smithi157 ceph-mon[33359]: pgmap v206: 340 pgs: 9 active+clean+snaptrim, 40 unknown, 6 active+clean+snaptrim_wait, 285 active+clean; 610 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s wr, 1 op/s 2024-03-11T20:55:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:36 smithi157 ceph-mon[33359]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi129-82555-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi129-82555-28"}]': finished 2024-03-11T20:55:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:36 smithi157 ceph-mon[33359]: osdmap e169: 8 total, 8 up, 8 in 2024-03-11T20:55:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:36.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:37.688 INFO:tasks.workunit.client.0.smithi129.stdout:P (67939 ms total) 2024-03-11T20:55:37.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2024-03-11T20:55:37.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-03-11T20:55:37.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-03-11T20:55:37.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (117 ms) 2024-03-11T20:55:37.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (117 ms total) 2024-03-11T20:55:37.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2024-03-11T20:55:37.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-03-11T20:55:37.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-03-11T20:55:37.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (35 ms) 2024-03-11T20:55:37.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-03-11T20:55:37.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (13 ms) 2024-03-11T20:55:37.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (48 ms total) 2024-03-11T20:55:37.689 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2024-03-11T20:55:37.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-03-11T20:55:37.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-03-11T20:55:37.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (28 ms) 2024-03-11T20:55:37.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-03-11T20:55:37.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (4 ms) 2024-03-11T20:55:37.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (32 ms total) 2024-03-11T20:55:37.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2024-03-11T20:55:37.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-03-11T20:55:37.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-03-11T20:55:37.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (50 ms) 2024-03-11T20:55:37.690 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-03-11T20:55:37.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (7 ms) 2024-03-11T20:55:37.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (57 ms total) 2024-03-11T20:55:37.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2024-03-11T20:55:37.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-03-11T20:55:37.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-03-11T20:55:37.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1019 ms) 2024-03-11T20:55:37.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1019 ms total) 2024-03-11T20:55:37.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: 2024-03-11T20:55:37.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-03-11T20:55:37.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (103931 ms total) 2024-03-11T20:55:37.691 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-03-11T20:55:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36"}]': finished 2024-03-11T20:55:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[30184]: osdmap e170: 8 total, 8 up, 8 in 2024-03-11T20:55:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3116834640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi129-82567-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi129-82567-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:37.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-11T20:55:37.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:37.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36"}]': finished 2024-03-11T20:55:37.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[42066]: osdmap e170: 8 total, 8 up, 8 in 2024-03-11T20:55:37.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3116834640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi129-82567-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:37.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:37.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi129-82567-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:37.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:37.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:37.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:37 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-11T20:55:37.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:37.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi129-82724-36"}]': finished 2024-03-11T20:55:37.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:37 smithi157 ceph-mon[33359]: osdmap e170: 8 total, 8 up, 8 in 2024-03-11T20:55:37.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3116834640' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi129-82567-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1258529228' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi129-82567-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi129-82724-36"}]: dispatch 2024-03-11T20:55:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:37.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:37 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-11T20:55:38.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[30184]: pgmap v209: 372 pgs: 32 creating+peering, 340 active+clean; 618 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s wr, 1 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T20:55:38.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi129-82567-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:38.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi129-82724-36"}]': finished 2024-03-11T20:55:38.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-11T20:55:38.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[30184]: osdmap e171: 8 total, 8 up, 8 in 2024-03-11T20:55:38.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:38.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:38.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:38.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:38.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:38.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[42066]: pgmap v209: 372 pgs: 32 creating+peering, 340 active+clean; 618 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s wr, 1 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T20:55:38.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi129-82567-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:38.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi129-82724-36"}]': finished 2024-03-11T20:55:38.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-11T20:55:38.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[42066]: osdmap e171: 8 total, 8 up, 8 in 2024-03-11T20:55:38.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:38.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:38.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:38.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:38.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:38.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:38 smithi157 ceph-mon[33359]: pgmap v209: 372 pgs: 32 creating+peering, 340 active+clean; 618 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s wr, 1 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T20:55:38.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi129-82567-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:38.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi129-82724-36"}]': finished 2024-03-11T20:55:38.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:38 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-11T20:55:38.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:38 smithi157 ceph-mon[33359]: osdmap e171: 8 total, 8 up, 8 in 2024-03-11T20:55:38.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:38.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:38.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:38.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:38.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:39.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi129-82555-28"}]': finished 2024-03-11T20:55:39.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:55:39.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: osdmap e172: 8 total, 8 up, 8 in 2024-03-11T20:55:39.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:39.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-15"}]: dispatch 2024-03-11T20:55:39.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:39.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-15"}]: dispatch 2024-03-11T20:55:39.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:39.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-11T20:55:39.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi129-82555-28"}]': finished 2024-03-11T20:55:39.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-15"}]': finished 2024-03-11T20:55:39.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-11T20:55:39.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: osdmap e173: 8 total, 8 up, 8 in 2024-03-11T20:55:39.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-15", "mode": "writeback"}]: dispatch 2024-03-11T20:55:39.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2947152175' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi129-82567-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:39.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-15", "mode": "writeback"}]: dispatch 2024-03-11T20:55:39.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi129-82567-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:39.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]: dispatch 2024-03-11T20:55:39.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi129-82555-28"}]': finished 2024-03-11T20:55:39.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:55:39.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: osdmap e172: 8 total, 8 up, 8 in 2024-03-11T20:55:39.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:39.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-15"}]: dispatch 2024-03-11T20:55:39.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:39.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-15"}]: dispatch 2024-03-11T20:55:39.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:39.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-11T20:55:39.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi129-82555-28"}]': finished 2024-03-11T20:55:39.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-15"}]': finished 2024-03-11T20:55:39.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-11T20:55:39.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: osdmap e173: 8 total, 8 up, 8 in 2024-03-11T20:55:39.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-15", "mode": "writeback"}]: dispatch 2024-03-11T20:55:39.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2947152175' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi129-82567-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:39.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-15", "mode": "writeback"}]: dispatch 2024-03-11T20:55:39.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi129-82567-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:39.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]: dispatch 2024-03-11T20:55:39.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi129-82555-28"}]': finished 2024-03-11T20:55:39.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:55:39.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: osdmap e172: 8 total, 8 up, 8 in 2024-03-11T20:55:39.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3830288215' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:39.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-15"}]: dispatch 2024-03-11T20:55:39.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi129-82555-28"}]: dispatch 2024-03-11T20:55:39.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-15"}]: dispatch 2024-03-11T20:55:39.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:39.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-11T20:55:39.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi129-82555-28"}]': finished 2024-03-11T20:55:39.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-15"}]': finished 2024-03-11T20:55:39.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi129-82637-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-11T20:55:39.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: osdmap e173: 8 total, 8 up, 8 in 2024-03-11T20:55:39.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-15", "mode": "writeback"}]: dispatch 2024-03-11T20:55:39.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2947152175' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi129-82567-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:39.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-15", "mode": "writeback"}]: dispatch 2024-03-11T20:55:39.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi129-82567-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:39.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]: dispatch 2024-03-11T20:55:40.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: pgmap v212: 332 pgs: 332 active+clean; 618 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T20:55:40.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi129-82555-29"}]: dispatch 2024-03-11T20:55:40.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi129-82555-29"}]: dispatch 2024-03-11T20:55:40.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi129-82555-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:40.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:40.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T20:55:40.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T20:55:40.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:55:40.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:55:40.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-15", "mode": "writeback"}]': finished 2024-03-11T20:55:40.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi129-82567-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:40.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]': finished 2024-03-11T20:55:40.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi129-82555-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:40.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]': finished 2024-03-11T20:55:40.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: osdmap e174: 8 total, 8 up, 8 in 2024-03-11T20:55:40.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:40.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]: dispatch 2024-03-11T20:55:40.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi129-82555-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi129-82555-29"}]: dispatch 2024-03-11T20:55:40.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:40.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T20:55:40.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T20:55:40.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: pgmap v212: 332 pgs: 332 active+clean; 618 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T20:55:40.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi129-82555-29"}]: dispatch 2024-03-11T20:55:40.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi129-82555-29"}]: dispatch 2024-03-11T20:55:40.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi129-82555-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:40.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:40.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T20:55:40.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T20:55:40.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:55:40.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:55:40.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-15", "mode": "writeback"}]': finished 2024-03-11T20:55:40.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi129-82567-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:40.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]': finished 2024-03-11T20:55:40.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi129-82555-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:40.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]': finished 2024-03-11T20:55:40.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: osdmap e174: 8 total, 8 up, 8 in 2024-03-11T20:55:40.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:40.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]: dispatch 2024-03-11T20:55:40.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi129-82555-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi129-82555-29"}]: dispatch 2024-03-11T20:55:40.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:40.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T20:55:40.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T20:55:40.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: pgmap v212: 332 pgs: 332 active+clean; 618 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-03-11T20:55:40.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi129-82555-29"}]: dispatch 2024-03-11T20:55:40.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi129-82555-29"}]: dispatch 2024-03-11T20:55:40.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi129-82555-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:40.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T20:55:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]: dispatch 2024-03-11T20:55:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:55:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:55:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-15", "mode": "writeback"}]': finished 2024-03-11T20:55:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi129-82567-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]': finished 2024-03-11T20:55:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi129-82555-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pg_num","val":"11"}]': finished 2024-03-11T20:55:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: osdmap e174: 8 total, 8 up, 8 in 2024-03-11T20:55:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]: dispatch 2024-03-11T20:55:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi129-82555-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi129-82555-29"}]: dispatch 2024-03-11T20:55:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:40.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T20:55:40.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-11T20:55:41.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-03-11T20:55:41.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-03-11T20:55:41.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-03-11T20:55:41.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-03-11T20:55:41.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-03-11T20:55:41.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (8597 ms) 2024-03-11T20:55:41.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-03-11T20:55:41.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (19095 ms) 2024-03-11T20:55:41.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-03-11T20:55:41.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6419 ms) 2024-03-11T20:55:41.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-03-11T20:55:41.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2863 ms) 2024-03-11T20:55:41.919 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-03-11T20:55:41.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3997 ms) 2024-03-11T20:55:41.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-03-11T20:55:41.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3009 ms) 2024-03-11T20:55:41.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-03-11T20:55:41.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (4013 ms) 2024-03-11T20:55:41.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-03-11T20:55:41.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3005 ms) 2024-03-11T20:55:41.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-03-11T20:55:41.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2999 ms) 2024-03-11T20:55:41.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-03-11T20:55:41.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2996 ms) 2024-03-11T20:55:41.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-03-11T20:55:41.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3251 ms) 2024-03-11T20:55:41.920 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-03-11T20:55:41.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2764 ms) 2024-03-11T20:55:41.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-03-11T20:55:41.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3002 ms) 2024-03-11T20:55:41.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-03-11T20:55:41.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3023 ms) 2024-03-11T20:55:41.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-03-11T20:55:41.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3007 ms) 2024-03-11T20:55:41.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-03-11T20:55:41.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3007 ms) 2024-03-11T20:55:41.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-03-11T20:55:41.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3008 ms) 2024-03-11T20:55:41.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-03-11T20:55:41.921 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2942 ms) 2024-03-11T20:55:41.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-03-11T20:55:41.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3017 ms) 2024-03-11T20:55:41.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-03-11T20:55:41.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2983 ms) 2024-03-11T20:55:41.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-03-11T20:55:41.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3010 ms) 2024-03-11T20:55:41.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-03-11T20:55:41.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3009 ms) 2024-03-11T20:55:41.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-03-11T20:55:41.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3019 ms) 2024-03-11T20:55:41.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-03-11T20:55:41.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3248 ms) 2024-03-11T20:55:41.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-03-11T20:55:41.922 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2726 ms) 2024-03-11T20:55:41.923 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-03-11T20:55:41.923 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3015 ms) 2024-03-11T20:55:41.923 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-03-11T20:55:41.926 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAiohotsECPP.SnapRemovePP (2008 ms) 2024-03-11T20:55:41.926 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-03-11T20:55:41.926 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2025 ms) 2024-03-11T20:55:41.926 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-03-11T20:55:41.927 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1970 ms) 2024-03-11T20:55:41.927 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8943 ms total) 2024-03-11T20:55:41.927 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: 2024-03-11T20:55:41.927 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-03-11T20:55:41.927 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-03-11T20:55:41.927 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4054 ms) 2024-03-11T20:55:41.927 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-03-11T20:55:41.927 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (4181 ms) 2024-03-11T20:55:41.927 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-03-11T20:55:41.927 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3770 ms) 2024-03-11T20:55:41.927 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12005 ms total) 2024-03-11T20:55:41.927 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: 2024-03-11T20:55:41.927 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-03-11T20:55:41.927 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (108143 ms total) 2024-03-11T20:55:41.927 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-03-11T20:55:41.928 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-03-11T20:55:41.928 INFO:tasks.workunit.client.0.smithi129.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-11T20:55:42.228 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:55:42.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:55:42.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:55:42.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:41 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]': finished 2024-03-11T20:55:42.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:55:42.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:55:42.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:55:42.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:41 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]': finished 2024-03-11T20:55:42.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:55:42.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:55:42.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:55:42.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:41 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4171864990' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi129-82913-21"}]': finished 2024-03-11T20:55:43.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: pgmap v215: 355 pgs: 32 unknown, 323 active+clean; 585 KiB data, 563 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:55:43.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]': finished 2024-03-11T20:55:43.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:43.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: osdmap e175: 8 total, 8 up, 8 in 2024-03-11T20:55:43.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:43.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:43.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:43.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi129-82555-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi129-82555-29"}]': finished 2024-03-11T20:55:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:55:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: osdmap e176: 8 total, 8 up, 8 in 2024-03-11T20:55:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15"}]: dispatch 2024-03-11T20:55:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1215871271' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi129-82567-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15"}]: dispatch 2024-03-11T20:55:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi129-82567-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi129-82637-2"}]: dispatch 2024-03-11T20:55:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi129-82637-2"}]: dispatch 2024-03-11T20:55:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi129-82637-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:43.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: pgmap v215: 355 pgs: 32 unknown, 323 active+clean; 585 KiB data, 563 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:55:43.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]': finished 2024-03-11T20:55:43.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:43.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: osdmap e175: 8 total, 8 up, 8 in 2024-03-11T20:55:43.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:43.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:43.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:43.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi129-82555-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi129-82555-29"}]': finished 2024-03-11T20:55:43.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:55:43.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: osdmap e176: 8 total, 8 up, 8 in 2024-03-11T20:55:43.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15"}]: dispatch 2024-03-11T20:55:43.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1215871271' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi129-82567-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:43.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15"}]: dispatch 2024-03-11T20:55:43.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi129-82567-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:43.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi129-82637-2"}]: dispatch 2024-03-11T20:55:43.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi129-82637-2"}]: dispatch 2024-03-11T20:55:43.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi129-82637-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:43.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: pgmap v215: 355 pgs: 32 unknown, 323 active+clean; 585 KiB data, 563 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:55:43.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi129-82637-1","var":"pgp_num","val":"11"}]': finished 2024-03-11T20:55:43.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4193314441' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-11T20:55:43.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: osdmap e175: 8 total, 8 up, 8 in 2024-03-11T20:55:43.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:43.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:43.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:43.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi129-82555-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi129-82555-29"}]': finished 2024-03-11T20:55:43.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:55:43.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: osdmap e176: 8 total, 8 up, 8 in 2024-03-11T20:55:43.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15"}]: dispatch 2024-03-11T20:55:43.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1215871271' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi129-82567-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:43.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15"}]: dispatch 2024-03-11T20:55:43.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi129-82567-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:43.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi129-82637-2"}]: dispatch 2024-03-11T20:55:43.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi129-82637-2"}]: dispatch 2024-03-11T20:55:43.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi129-82637-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:43.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:55:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:55:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:43 smithi129 ceph-mon[30184]: pgmap v218: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:55:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:43 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:43 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:55:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15"}]': finished 2024-03-11T20:55:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi129-82567-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi129-82637-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:43 smithi129 ceph-mon[30184]: osdmap e177: 8 total, 8 up, 8 in 2024-03-11T20:55:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi129-82637-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi129-82637-2"}]: dispatch 2024-03-11T20:55:44.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:43 smithi129 ceph-mon[42066]: pgmap v218: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:55:44.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:43 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:44.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:43 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:55:44.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15"}]': finished 2024-03-11T20:55:44.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi129-82567-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:44.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi129-82637-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:44.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:43 smithi129 ceph-mon[42066]: osdmap e177: 8 total, 8 up, 8 in 2024-03-11T20:55:44.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi129-82637-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi129-82637-2"}]: dispatch 2024-03-11T20:55:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:43 smithi157 ceph-mon[33359]: pgmap v218: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:55:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:43 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:43 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:55:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-15"}]': finished 2024-03-11T20:55:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi129-82567-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi129-82637-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:43 smithi157 ceph-mon[33359]: osdmap e177: 8 total, 8 up, 8 in 2024-03-11T20:55:44.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi129-82637-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi129-82637-2"}]: dispatch 2024-03-11T20:55:45.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:45 smithi129 ceph-mon[30184]: osdmap e178: 8 total, 8 up, 8 in 2024-03-11T20:55:45.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi129-82555-29"}]: dispatch 2024-03-11T20:55:45.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:45 smithi129 ceph-mon[42066]: osdmap e178: 8 total, 8 up, 8 in 2024-03-11T20:55:45.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi129-82555-29"}]: dispatch 2024-03-11T20:55:45.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:45 smithi157 ceph-mon[33359]: osdmap e178: 8 total, 8 up, 8 in 2024-03-11T20:55:45.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi129-82555-29"}]: dispatch 2024-03-11T20:55:46.478 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:55:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:55:46.768 INFO:tasks.workunit.client.0.smithi129.stdout:d=5 2024-03-11T20:55:46.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:863748b0:::15:head 2024-03-11T20:55:46.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=10:863748b0:::15:head expected=10:863748b0:::15:head 2024-03-11T20:55:46.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:863748b0:::15:head -> 15 2024-03-11T20:55:46.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=15 expected=15 2024-03-11T20:55:46.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:6cac518f:::0:head 2024-03-11T20:55:46.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=10:6cac518f:::0:head expected=10:6cac518f:::0:head 2024-03-11T20:55:46.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:6cac518f:::0:head -> 0 2024-03-11T20:55:46.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=0 expected=0 2024-03-11T20:55:46.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:62a1935d:::14:head 2024-03-11T20:55:46.768 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=10:62a1935d:::14:head expected=10:62a1935d:::14:head 2024-03-11T20:55:46.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:62a1935d:::14:head -> 14 2024-03-11T20:55:46.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=14 expected=14 2024-03-11T20:55:46.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:5c6b0b28:::7:head 2024-03-11T20:55:46.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=10:5c6b0b28:::7:head expected=10:5c6b0b28:::7:head 2024-03-11T20:55:46.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:5c6b0b28:::7:head -> 7 2024-03-11T20:55:46.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=7 expected=7 2024-03-11T20:55:46.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:566253c9:::13:head 2024-03-11T20:55:46.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=10:566253c9:::13:head expected=10:566253c9:::13:head 2024-03-11T20:55:46.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:566253c9:::13:head -> 13 2024-03-11T20:55:46.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=13 expected=13 2024-03-11T20:55:46.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:52ea6a34:::10:head 2024-03-11T20:55:46.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=10:52ea6a34:::10:head expected=10:52ea6a34:::10:head 2024-03-11T20:55:46.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:52ea6a34:::10:head -> 10 2024-03-11T20:55:46.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=10 expected=10 2024-03-11T20:55:46.769 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:02547ec2:::1:head 2024-03-11T20:55:46.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=10:02547ec2:::1:head expected=10:02547ec2:::1:head 2024-03-11T20:55:46.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:02547ec2:::1:head -> 1 2024-03-11T20:55:46.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=1 expected=1 2024-03-11T20:55:46.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : seek to 10:89d3ae78:::11:head 2024-03-11T20:55:46.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : cursor()=10:89d3ae78:::11:head expected=10:89d3ae78:::11:head 2024-03-11T20:55:46.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: > 10:89d3ae78:::11:head -> 11 2024-03-11T20:55:46.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: : entry=11 expected=11 2024-03-11T20:55:46.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (226 ms) 2024-03-11T20:55:46.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-03-11T20:55:46.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (76126 ms) 2024-03-11T20:55:46.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-03-11T20:55:46.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: split 0/5 -> MIN 10:33333333::::head 2024-03-11T20:55:46.770 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: split 1/5 -> 10:33333333::::head 10:66666666::::head 2024-03-11T20:55:46.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: split 2/5 -> 10:66666666::::head 10:99999999::::head 2024-03-11T20:55:46.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: split 3/5 -> 10:99999999::::head 10:cccccccc::::head 2024-03-11T20:55:46.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: split 4/5 -> 10:cccccccc::::head MAX 2024-03-11T20:55:46.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (22013 ms) 2024-03-11T20:55:46.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 7 tests from LibRadosList (98980 ms total) 2024-03-11T20:55:46.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 2024-03-11T20:55:46.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-03-11T20:55:46.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-03-11T20:55:46.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1056 ms) 2024-03-11T20:55:46.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-03-11T20:55:46.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo1,foo2,foo3 2024-03-11T20:55:46.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo1 2024-03-11T20:55:46.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo2 2024-03-11T20:55:46.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo3 2024-03-11T20:55:46.771 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo1,foo4,foo5 2024-03-11T20:55:46.772 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo4 2024-03-11T20:55:46.772 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo5 2024-03-11T20:55:46.772 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo1 2024-03-11T20:55:46.772 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset foo6,foo7 2024-03-11T20:55:46.772 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo7 2024-03-11T20:55:46.772 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: foo6 2024-03-11T20:55:46.772 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-11T20:55:46.772 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo4 2024-03-11T20:55:46.772 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo5 2024-03-11T20:55:46.772 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns2:foo7 2024-03-11T20:55:46.772 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns2:foo6 2024-03-11T20:55:46.772 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: ns1:foo1 2024-03-11T20:55:46.772 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo1 2024-03-11T20:55:46.772 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo2 2024-03-11T20:55:46.773 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: :foo3 2024-03-11T20:55:46.773 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (31 ms) 2024-03-11T20:55:46.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[42066]: pgmap v221: 260 pgs: 260 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:55:46.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi129-82637-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi129-82637-2"}]': finished 2024-03-11T20:55:46.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi129-82555-29"}]': finished 2024-03-11T20:55:46.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[42066]: osdmap e179: 8 total, 8 up, 8 in 2024-03-11T20:55:46.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi129-82555-29"}]: dispatch 2024-03-11T20:55:46.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:46.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/241446799' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi129-82567-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:46.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:46.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi129-82567-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:46.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[30184]: pgmap v221: 260 pgs: 260 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:55:46.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi129-82637-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi129-82637-2"}]': finished 2024-03-11T20:55:46.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi129-82555-29"}]': finished 2024-03-11T20:55:46.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[30184]: osdmap e179: 8 total, 8 up, 8 in 2024-03-11T20:55:46.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi129-82555-29"}]: dispatch 2024-03-11T20:55:46.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:46.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/241446799' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi129-82567-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:46.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:46.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:46 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi129-82567-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:46.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:46 smithi157 ceph-mon[33359]: pgmap v221: 260 pgs: 260 active+clean; 584 KiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:55:46.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:46 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi129-82637-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi129-82637-2"}]': finished 2024-03-11T20:55:46.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:46 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi129-82555-29"}]': finished 2024-03-11T20:55:46.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:46 smithi157 ceph-mon[33359]: osdmap e179: 8 total, 8 up, 8 in 2024-03-11T20:55:46.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:46 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi129-82555-29"}]: dispatch 2024-03-11T20:55:46.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:46 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:46.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:46 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/241446799' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi129-82567-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:46.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:46 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:46.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:46 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi129-82567-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:47.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi129-82555-29"}]': finished 2024-03-11T20:55:47.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:47.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi129-82567-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:47.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[30184]: osdmap e180: 8 total, 8 up, 8 in 2024-03-11T20:55:47.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi129-82555-30"}]: dispatch 2024-03-11T20:55:47.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi129-82555-30"}]: dispatch 2024-03-11T20:55:47.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi129-82555-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:47.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:47.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:47.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi129-82555-29"}]': finished 2024-03-11T20:55:47.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:47.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi129-82567-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:47.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[42066]: osdmap e180: 8 total, 8 up, 8 in 2024-03-11T20:55:47.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi129-82555-30"}]: dispatch 2024-03-11T20:55:47.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi129-82555-30"}]: dispatch 2024-03-11T20:55:47.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi129-82555-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:47.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:47.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:47 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:47.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732752469' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi129-82555-29"}]': finished 2024-03-11T20:55:47.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:47 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:47.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:47 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi129-82567-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:47.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:47 smithi157 ceph-mon[33359]: osdmap e180: 8 total, 8 up, 8 in 2024-03-11T20:55:47.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi129-82555-30"}]: dispatch 2024-03-11T20:55:47.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi129-82555-30"}]: dispatch 2024-03-11T20:55:47.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi129-82555-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:47.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:47.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:47 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[30184]: pgmap v224: 332 pgs: 332 active+clean; 585 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-03-11T20:55:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi129-82555-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:55:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[30184]: osdmap e181: 8 total, 8 up, 8 in 2024-03-11T20:55:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-17"}]: dispatch 2024-03-11T20:55:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi129-82555-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi129-82555-30"}]: dispatch 2024-03-11T20:55:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-17"}]: dispatch 2024-03-11T20:55:49.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi129-82637-2"}]: dispatch 2024-03-11T20:55:49.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:55:49.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[42066]: pgmap v224: 332 pgs: 332 active+clean; 585 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-03-11T20:55:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi129-82555-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:55:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[42066]: osdmap e181: 8 total, 8 up, 8 in 2024-03-11T20:55:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-17"}]: dispatch 2024-03-11T20:55:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi129-82555-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi129-82555-30"}]: dispatch 2024-03-11T20:55:49.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-17"}]: dispatch 2024-03-11T20:55:49.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi129-82637-2"}]: dispatch 2024-03-11T20:55:49.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:55:49.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:48 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:49.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:48 smithi157 ceph-mon[33359]: pgmap v224: 332 pgs: 332 active+clean; 585 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-03-11T20:55:49.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:48 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:49.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi129-82555-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:49.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:48 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:55:49.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:48 smithi157 ceph-mon[33359]: osdmap e181: 8 total, 8 up, 8 in 2024-03-11T20:55:49.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-17"}]: dispatch 2024-03-11T20:55:49.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi129-82555-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi129-82555-30"}]: dispatch 2024-03-11T20:55:49.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:48 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-17"}]: dispatch 2024-03-11T20:55:49.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi129-82637-2"}]: dispatch 2024-03-11T20:55:49.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:48 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:55:49.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:48 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:55:50.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-17"}]': finished 2024-03-11T20:55:50.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi129-82637-2"}]': finished 2024-03-11T20:55:50.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-17", "mode": "writeback"}]: dispatch 2024-03-11T20:55:50.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:49 smithi157 ceph-mon[33359]: pgmap v227: 324 pgs: 32 unknown, 292 active+clean; 584 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:55:50.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:49 smithi157 ceph-mon[33359]: osdmap e182: 8 total, 8 up, 8 in 2024-03-11T20:55:50.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1039741762' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi129-82567-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:50.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi129-82637-2"}]: dispatch 2024-03-11T20:55:50.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-17", "mode": "writeback"}]: dispatch 2024-03-11T20:55:50.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi129-82567-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:50.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:49 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:55:50.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi129-82555-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi129-82555-30"}]': finished 2024-03-11T20:55:50.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi129-82637-2"}]': finished 2024-03-11T20:55:50.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-17", "mode": "writeback"}]': finished 2024-03-11T20:55:50.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi129-82567-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:50.239 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:49 smithi157 ceph-mon[33359]: osdmap e183: 8 total, 8 up, 8 in 2024-03-11T20:55:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-17"}]': finished 2024-03-11T20:55:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi129-82637-2"}]': finished 2024-03-11T20:55:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-17", "mode": "writeback"}]: dispatch 2024-03-11T20:55:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[30184]: pgmap v227: 324 pgs: 32 unknown, 292 active+clean; 584 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:55:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[30184]: osdmap e182: 8 total, 8 up, 8 in 2024-03-11T20:55:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1039741762' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi129-82567-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi129-82637-2"}]: dispatch 2024-03-11T20:55:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-17", "mode": "writeback"}]: dispatch 2024-03-11T20:55:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi129-82567-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:55:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi129-82555-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi129-82555-30"}]': finished 2024-03-11T20:55:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi129-82637-2"}]': finished 2024-03-11T20:55:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-17", "mode": "writeback"}]': finished 2024-03-11T20:55:50.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi129-82567-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:50.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[30184]: osdmap e183: 8 total, 8 up, 8 in 2024-03-11T20:55:50.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-17"}]': finished 2024-03-11T20:55:50.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi129-82637-2"}]': finished 2024-03-11T20:55:50.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-17", "mode": "writeback"}]: dispatch 2024-03-11T20:55:50.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[42066]: pgmap v227: 324 pgs: 32 unknown, 292 active+clean; 584 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:55:50.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[42066]: osdmap e182: 8 total, 8 up, 8 in 2024-03-11T20:55:50.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1039741762' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi129-82567-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:50.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi129-82637-2"}]: dispatch 2024-03-11T20:55:50.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-17", "mode": "writeback"}]: dispatch 2024-03-11T20:55:50.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi129-82567-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:50.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:55:50.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi129-82555-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi129-82555-30"}]': finished 2024-03-11T20:55:50.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2455334614' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi129-82637-2"}]': finished 2024-03-11T20:55:50.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-17", "mode": "writeback"}]': finished 2024-03-11T20:55:50.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi129-82567-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:50.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:49 smithi129 ceph-mon[42066]: osdmap e183: 8 total, 8 up, 8 in 2024-03-11T20:55:51.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:51.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:51.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:55:51.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:50 smithi157 ceph-mon[33359]: osdmap e184: 8 total, 8 up, 8 in 2024-03-11T20:55:51.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17"}]: dispatch 2024-03-11T20:55:51.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17"}]: dispatch 2024-03-11T20:55:51.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1732412483' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82637-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:51.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82637-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:51.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:51.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:51.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:55:51.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:50 smithi129 ceph-mon[30184]: osdmap e184: 8 total, 8 up, 8 in 2024-03-11T20:55:51.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17"}]: dispatch 2024-03-11T20:55:51.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17"}]: dispatch 2024-03-11T20:55:51.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1732412483' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82637-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:51.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82637-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:51.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:51.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:51.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:55:51.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:50 smithi129 ceph-mon[42066]: osdmap e184: 8 total, 8 up, 8 in 2024-03-11T20:55:51.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17"}]: dispatch 2024-03-11T20:55:51.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17"}]: dispatch 2024-03-11T20:55:51.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1732412483' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82637-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:51.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82637-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:52.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:51 smithi157 ceph-mon[33359]: pgmap v230: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 573 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:55:52.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:51 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:55:52.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:51 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17"}]': finished 2024-03-11T20:55:52.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:51 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82637-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:52.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:51 smithi157 ceph-mon[33359]: osdmap e185: 8 total, 8 up, 8 in 2024-03-11T20:55:52.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2904225852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi129-82567-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:52.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi129-82555-30"}]: dispatch 2024-03-11T20:55:52.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1732412483' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi129-82637-3","pool2":"test-rados-api-smithi129-82637-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-11T20:55:52.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:51 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi129-82637-3","pool2":"test-rados-api-smithi129-82637-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-11T20:55:52.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[30184]: pgmap v230: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 573 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:55:52.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:55:52.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17"}]': finished 2024-03-11T20:55:52.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82637-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:52.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[30184]: osdmap e185: 8 total, 8 up, 8 in 2024-03-11T20:55:52.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2904225852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi129-82567-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:52.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi129-82555-30"}]: dispatch 2024-03-11T20:55:52.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1732412483' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi129-82637-3","pool2":"test-rados-api-smithi129-82637-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-11T20:55:52.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi129-82637-3","pool2":"test-rados-api-smithi129-82637-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-11T20:55:52.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[42066]: pgmap v230: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 573 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:55:52.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:55:52.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-17"}]': finished 2024-03-11T20:55:52.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82637-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:52.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[42066]: osdmap e185: 8 total, 8 up, 8 in 2024-03-11T20:55:52.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2904225852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi129-82567-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:52.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi129-82555-30"}]: dispatch 2024-03-11T20:55:52.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1732412483' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi129-82637-3","pool2":"test-rados-api-smithi129-82637-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-11T20:55:52.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:51 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi129-82637-3","pool2":"test-rados-api-smithi129-82637-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-11T20:55:52.714 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-03-11T20:55:52.714 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 1 0 2024-03-11T20:55:52.714 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 10 0 2024-03-11T20:55:52.714 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 13 0 2024-03-11T20:55:52.714 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 7 0 2024-03-11T20:55:52.714 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 14 0 2024-03-11T20:55:52.714 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 0 0 2024-03-11T20:55:52.714 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 15 0 2024-03-11T20:55:52.714 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 11 0 2024-03-11T20:55:52.714 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 5 0 2024-03-11T20:55:52.714 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 8 0 2024-03-11T20:55:52.715 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 6 0 2024-03-11T20:55:52.715 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 3 0 2024-03-11T20:55:52.715 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 4 0 2024-03-11T20:55:52.715 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 12 0 2024-03-11T20:55:52.715 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 9 0 2024-03-11T20:55:52.715 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 2 0 2024-03-11T20:55:52.715 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-11T20:55:52.715 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (44 ms) 2024-03-11T20:55:52.715 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 3 tests from LibRadosListEC (1131 ms total) 2024-03-11T20:55:52.715 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 2024-03-11T20:55:52.715 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-03-11T20:55:52.715 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-03-11T20:55:52.715 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2999 ms) 2024-03-11T20:55:52.715 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] 1 test from LibRadosListNP (2999 ms total) 2024-03-11T20:55:52.715 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: 2024-03-11T20:55:52.716 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [----------] Global test environment tear-down 2024-03-11T20:55:52.716 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [==========] 11 tests from 3 test suites ran. (119004 ms total) 2024-03-11T20:55:52.716 INFO:tasks.workunit.client.0.smithi129.stdout: api_list: [ PASSED ] 11 tests. 2024-03-11T20:55:53.705 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:55:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:55:53.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2904225852' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi129-82567-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:53.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi129-82555-30"}]': finished 2024-03-11T20:55:53.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi129-82637-3","pool2":"test-rados-api-smithi129-82637-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-11T20:55:53.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[42066]: osdmap e186: 8 total, 8 up, 8 in 2024-03-11T20:55:53.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi129-82555-30"}]: dispatch 2024-03-11T20:55:53.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2904225852' entity='client.admin' cmd=[{ 2024-03-11T20:55:53.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[42066]: "prefix": "osd pool set", 2024-03-11T20:55:53.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[42066]: "pool": "PoolEIOFlag_smithi129-82567-33", 2024-03-11T20:55:53.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[42066]: "var": "eio", 2024-03-11T20:55:53.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[42066]: "val": "true" 2024-03-11T20:55:53.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[42066]: }]: dispatch 2024-03-11T20:55:53.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2904225852' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi129-82567-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:53.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi129-82555-30"}]': finished 2024-03-11T20:55:53.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi129-82637-3","pool2":"test-rados-api-smithi129-82637-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-11T20:55:53.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[30184]: osdmap e186: 8 total, 8 up, 8 in 2024-03-11T20:55:53.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi129-82555-30"}]: dispatch 2024-03-11T20:55:53.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2904225852' entity='client.admin' cmd=[{ 2024-03-11T20:55:53.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[30184]: "prefix": "osd pool set", 2024-03-11T20:55:53.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[30184]: "pool": "PoolEIOFlag_smithi129-82567-33", 2024-03-11T20:55:53.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[30184]: "var": "eio", 2024-03-11T20:55:53.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[30184]: "val": "true" 2024-03-11T20:55:53.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:53 smithi129 ceph-mon[30184]: }]: dispatch 2024-03-11T20:55:53.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:53 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2904225852' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi129-82567-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:53.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:53 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi129-82555-30"}]': finished 2024-03-11T20:55:53.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:53 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi129-82637-3","pool2":"test-rados-api-smithi129-82637-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-11T20:55:53.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:53 smithi157 ceph-mon[33359]: osdmap e186: 8 total, 8 up, 8 in 2024-03-11T20:55:53.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:53 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi129-82555-30"}]: dispatch 2024-03-11T20:55:53.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:53 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2904225852' entity='client.admin' cmd=[{ 2024-03-11T20:55:53.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:53 smithi157 ceph-mon[33359]: "prefix": "osd pool set", 2024-03-11T20:55:53.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:53 smithi157 ceph-mon[33359]: "pool": "PoolEIOFlag_smithi129-82567-33", 2024-03-11T20:55:53.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:53 smithi157 ceph-mon[33359]: "var": "eio", 2024-03-11T20:55:53.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:53 smithi157 ceph-mon[33359]: "val": "true" 2024-03-11T20:55:53.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:53 smithi157 ceph-mon[33359]: }]: dispatch 2024-03-11T20:55:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: pgmap v233: 292 pgs: 32 creating+peering, 260 active+clean; 584 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:55:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi129-82555-30"}]': finished 2024-03-11T20:55:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2904225852' entity='client.admin' cmd='[{ 2024-03-11T20:55:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: "prefix": "osd pool set", 2024-03-11T20:55:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: "pool": "PoolEIOFlag_smithi129-82567-33", 2024-03-11T20:55:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: "var": "eio", 2024-03-11T20:55:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: "val": "true" 2024-03-11T20:55:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: }]': finished 2024-03-11T20:55:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: osdmap e187: 8 total, 8 up, 8 in 2024-03-11T20:55:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi129-82555-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi129-82555-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: pgmap v233: 292 pgs: 32 creating+peering, 260 active+clean; 584 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:55:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi129-82555-30"}]': finished 2024-03-11T20:55:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2904225852' entity='client.admin' cmd='[{ 2024-03-11T20:55:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: "prefix": "osd pool set", 2024-03-11T20:55:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: "pool": "PoolEIOFlag_smithi129-82567-33", 2024-03-11T20:55:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: "var": "eio", 2024-03-11T20:55:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: "val": "true" 2024-03-11T20:55:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: }]': finished 2024-03-11T20:55:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: osdmap e187: 8 total, 8 up, 8 in 2024-03-11T20:55:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:54.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:54.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:54.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:54.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:54.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:54.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi129-82555-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:54.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi129-82555-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: pgmap v233: 292 pgs: 32 creating+peering, 260 active+clean; 584 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:55:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:55:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:55:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4225945559' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi129-82555-30"}]': finished 2024-03-11T20:55:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2904225852' entity='client.admin' cmd='[{ 2024-03-11T20:55:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: "prefix": "osd pool set", 2024-03-11T20:55:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: "pool": "PoolEIOFlag_smithi129-82567-33", 2024-03-11T20:55:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: "var": "eio", 2024-03-11T20:55:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: "val": "true" 2024-03-11T20:55:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: }]': finished 2024-03-11T20:55:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: osdmap e187: 8 total, 8 up, 8 in 2024-03-11T20:55:54.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:54.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:54.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:54.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:54.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:54.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:54.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi129-82555-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:54.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi129-82555-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:55:55.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:55.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi129-82555-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:55.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:55 smithi129 ceph-mon[30184]: osdmap e188: 8 total, 8 up, 8 in 2024-03-11T20:55:55.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi129-82555-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:55.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi129-82555-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:55.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:55.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:55.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:55.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi129-82555-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:55.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:55 smithi129 ceph-mon[42066]: osdmap e188: 8 total, 8 up, 8 in 2024-03-11T20:55:55.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi129-82555-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:55.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi129-82555-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:55.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:55.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:55.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:55.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi129-82555-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:55:55.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:55 smithi157 ceph-mon[33359]: osdmap e188: 8 total, 8 up, 8 in 2024-03-11T20:55:55.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi129-82555-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:55.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi129-82555-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:55:55.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:55.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:55:56.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:55:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:55:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:55:56.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:56 smithi129 ceph-mon[30184]: pgmap v236: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:55:56.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:56 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:55:56.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-19"}]: dispatch 2024-03-11T20:55:56.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:56 smithi129 ceph-mon[30184]: osdmap e189: 8 total, 8 up, 8 in 2024-03-11T20:55:56.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2476413354' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi129-82567-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:56.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:56 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-19"}]: dispatch 2024-03-11T20:55:56.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:56 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi129-82567-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:56.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:56 smithi129 ceph-mon[42066]: pgmap v236: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:55:56.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:56 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:55:56.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-19"}]: dispatch 2024-03-11T20:55:56.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:56 smithi129 ceph-mon[42066]: osdmap e189: 8 total, 8 up, 8 in 2024-03-11T20:55:56.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2476413354' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi129-82567-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:56.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:56 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-19"}]: dispatch 2024-03-11T20:55:56.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:56 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi129-82567-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:56.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:56 smithi157 ceph-mon[33359]: pgmap v236: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 561 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:55:56.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:56 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:55:56.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-19"}]: dispatch 2024-03-11T20:55:56.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:56 smithi157 ceph-mon[33359]: osdmap e189: 8 total, 8 up, 8 in 2024-03-11T20:55:56.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2476413354' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi129-82567-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:56.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:56 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-19"}]: dispatch 2024-03-11T20:55:56.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:56 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi129-82567-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:55:57.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi129-82555-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi129-82555-31"}]': finished 2024-03-11T20:55:57.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-19"}]': finished 2024-03-11T20:55:57.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi129-82567-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:57.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:57 smithi157 ceph-mon[33359]: osdmap e190: 8 total, 8 up, 8 in 2024-03-11T20:55:57.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-19", "mode": "writeback"}]: dispatch 2024-03-11T20:55:57.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-19", "mode": "writeback"}]: dispatch 2024-03-11T20:55:57.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:57 smithi157 ceph-mon[33359]: pgmap v239: 332 pgs: 8 creating+peering, 1 active, 2 creating+activating, 321 active+clean; 584 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-11T20:55:58.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi129-82555-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi129-82555-31"}]': finished 2024-03-11T20:55:58.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-19"}]': finished 2024-03-11T20:55:58.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi129-82567-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:58.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:57 smithi129 ceph-mon[30184]: osdmap e190: 8 total, 8 up, 8 in 2024-03-11T20:55:58.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-19", "mode": "writeback"}]: dispatch 2024-03-11T20:55:58.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-19", "mode": "writeback"}]: dispatch 2024-03-11T20:55:58.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:57 smithi129 ceph-mon[30184]: pgmap v239: 332 pgs: 8 creating+peering, 1 active, 2 creating+activating, 321 active+clean; 584 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-11T20:55:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi129-82555-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi129-82555-31"}]': finished 2024-03-11T20:55:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-19"}]': finished 2024-03-11T20:55:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi129-82567-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:55:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:57 smithi129 ceph-mon[42066]: osdmap e190: 8 total, 8 up, 8 in 2024-03-11T20:55:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-19", "mode": "writeback"}]: dispatch 2024-03-11T20:55:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-19", "mode": "writeback"}]: dispatch 2024-03-11T20:55:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:57 smithi129 ceph-mon[42066]: pgmap v239: 332 pgs: 8 creating+peering, 1 active, 2 creating+activating, 321 active+clean; 584 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-11T20:55:58.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:58 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:55:58.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:58 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-19", "mode": "writeback"}]': finished 2024-03-11T20:55:58.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:58 smithi157 ceph-mon[33359]: osdmap e191: 8 total, 8 up, 8 in 2024-03-11T20:55:58.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:58.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:58 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:58 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:55:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:58 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-19", "mode": "writeback"}]': finished 2024-03-11T20:55:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:58 smithi129 ceph-mon[30184]: osdmap e191: 8 total, 8 up, 8 in 2024-03-11T20:55:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:58 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:59.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:58 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:55:59.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:58 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-19", "mode": "writeback"}]': finished 2024-03-11T20:55:59.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:58 smithi129 ceph-mon[42066]: osdmap e191: 8 total, 8 up, 8 in 2024-03-11T20:55:59.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:55:59.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:58 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:00.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:00.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[30184]: osdmap e192: 8 total, 8 up, 8 in 2024-03-11T20:56:00.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:56:00.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3425893796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi129-82567-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:00.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:56:00.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi129-82567-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:00.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19"}]: dispatch 2024-03-11T20:56:00.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19"}]: dispatch 2024-03-11T20:56:00.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[30184]: pgmap v242: 324 pgs: 32 unknown, 292 active+clean; 584 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:56:00.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:00.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:00.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[42066]: osdmap e192: 8 total, 8 up, 8 in 2024-03-11T20:56:00.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:56:00.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3425893796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi129-82567-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:00.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:56:00.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi129-82567-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:00.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19"}]: dispatch 2024-03-11T20:56:00.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19"}]: dispatch 2024-03-11T20:56:00.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[42066]: pgmap v242: 324 pgs: 32 unknown, 292 active+clean; 584 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:56:00.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:55:59 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:00.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:59 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:00.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:59 smithi157 ceph-mon[33359]: osdmap e192: 8 total, 8 up, 8 in 2024-03-11T20:56:00.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:56:00.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3425893796' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi129-82567-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:00.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:59 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:56:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:59 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi129-82567-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19"}]: dispatch 2024-03-11T20:56:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:59 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19"}]: dispatch 2024-03-11T20:56:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:59 smithi157 ceph-mon[33359]: pgmap v242: 324 pgs: 32 unknown, 292 active+clean; 584 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:56:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:55:59 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:01.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:00 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:01.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:00 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi129-82555-31"}]': finished 2024-03-11T20:56:01.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:00 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi129-82567-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:01.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:00 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19"}]': finished 2024-03-11T20:56:01.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:00 smithi129 ceph-mon[30184]: osdmap e193: 8 total, 8 up, 8 in 2024-03-11T20:56:01.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:00 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:56:01.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:00 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:56:01.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:00 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:01.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:00 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi129-82555-31"}]': finished 2024-03-11T20:56:01.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:00 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi129-82567-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:01.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:00 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19"}]': finished 2024-03-11T20:56:01.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:00 smithi129 ceph-mon[42066]: osdmap e193: 8 total, 8 up, 8 in 2024-03-11T20:56:01.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:00 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:56:01.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:00 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:56:01.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:00 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:01.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:00 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi129-82555-31"}]': finished 2024-03-11T20:56:01.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:00 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi129-82567-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:01.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:00 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-19"}]': finished 2024-03-11T20:56:01.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:00 smithi157 ceph-mon[33359]: osdmap e193: 8 total, 8 up, 8 in 2024-03-11T20:56:01.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:00 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1180320141' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:56:01.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:00 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi129-82555-31"}]: dispatch 2024-03-11T20:56:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi129-82555-31"}]': finished 2024-03-11T20:56:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[30184]: osdmap e194: 8 total, 8 up, 8 in 2024-03-11T20:56:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[30184]: pgmap v245: 260 pgs: 260 active+clean; 584 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.b", "id": [0, 4]}]: dispatch 2024-03-11T20:56:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:02.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.1a", "id": [0, 4]}]: dispatch 2024-03-11T20:56:02.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi129-82555-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:02.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi129-82555-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:02.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi129-82555-31"}]': finished 2024-03-11T20:56:02.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[42066]: osdmap e194: 8 total, 8 up, 8 in 2024-03-11T20:56:02.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:02.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[42066]: pgmap v245: 260 pgs: 260 active+clean; 584 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:02.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:02.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:02.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.b", "id": [0, 4]}]: dispatch 2024-03-11T20:56:02.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:02.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.1a", "id": [0, 4]}]: dispatch 2024-03-11T20:56:02.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi129-82555-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:02.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:01 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi129-82555-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:02.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:01 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi129-82555-31"}]': finished 2024-03-11T20:56:02.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:01 smithi157 ceph-mon[33359]: osdmap e194: 8 total, 8 up, 8 in 2024-03-11T20:56:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:01 smithi157 ceph-mon[33359]: pgmap v245: 260 pgs: 260 active+clean; 584 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:01 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.b", "id": [0, 4]}]: dispatch 2024-03-11T20:56:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:01 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.1a", "id": [0, 4]}]: dispatch 2024-03-11T20:56:02.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi129-82555-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:02.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:01 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi129-82555-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:03.382 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:03 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.b", "id": [0, 4]}]': finished 2024-03-11T20:56:03.382 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:03 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.1a", "id": [0, 4]}]': finished 2024-03-11T20:56:03.382 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi129-82555-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:03.382 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:03 smithi157 ceph-mon[33359]: osdmap e195: 8 total, 8 up, 8 in 2024-03-11T20:56:03.382 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi129-82555-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:03.383 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:03.383 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3360546170' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:03.383 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi129-82555-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:03.383 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:03.383 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:03.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.b", "id": [0, 4]}]': finished 2024-03-11T20:56:03.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.1a", "id": [0, 4]}]': finished 2024-03-11T20:56:03.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi129-82555-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:03.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[30184]: osdmap e195: 8 total, 8 up, 8 in 2024-03-11T20:56:03.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi129-82555-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:03.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:03.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3360546170' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:03.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi129-82555-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:03.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:03.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:03.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.b", "id": [0, 4]}]': finished 2024-03-11T20:56:03.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "46.1a", "id": [0, 4]}]': finished 2024-03-11T20:56:03.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi129-82555-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:03.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[42066]: osdmap e195: 8 total, 8 up, 8 in 2024-03-11T20:56:03.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi129-82555-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:03.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:03.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3360546170' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:03.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi129-82555-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:03.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:03.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:03.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:56:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:56:04.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:04.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:04.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[30184]: pgmap v248: 324 pgs: 13 creating+peering, 1 peering, 51 unknown, 259 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:56:04.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:04.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[30184]: osdmap e196: 8 total, 8 up, 8 in 2024-03-11T20:56:04.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:04.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:04.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:04.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi129-82555-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi129-82555-32"}]': finished 2024-03-11T20:56:04.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:04.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[30184]: osdmap e197: 8 total, 8 up, 8 in 2024-03-11T20:56:04.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-21"}]: dispatch 2024-03-11T20:56:04.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-21"}]: dispatch 2024-03-11T20:56:04.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2916558063' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:04.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:04.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:04.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:04.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[42066]: pgmap v248: 324 pgs: 13 creating+peering, 1 peering, 51 unknown, 259 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:56:04.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:04.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[42066]: osdmap e196: 8 total, 8 up, 8 in 2024-03-11T20:56:04.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:04.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:04.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:04.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi129-82555-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi129-82555-32"}]': finished 2024-03-11T20:56:04.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:04.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[42066]: osdmap e197: 8 total, 8 up, 8 in 2024-03-11T20:56:04.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-21"}]: dispatch 2024-03-11T20:56:04.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-21"}]: dispatch 2024-03-11T20:56:04.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2916558063' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:04.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:04.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:04.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:04.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:04 smithi157 ceph-mon[33359]: pgmap v248: 324 pgs: 13 creating+peering, 1 peering, 51 unknown, 259 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:56:04.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:04.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:04 smithi157 ceph-mon[33359]: osdmap e196: 8 total, 8 up, 8 in 2024-03-11T20:56:04.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:04.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:04.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:04 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:04.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi129-82555-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi129-82555-32"}]': finished 2024-03-11T20:56:04.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:04.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:04 smithi157 ceph-mon[33359]: osdmap e197: 8 total, 8 up, 8 in 2024-03-11T20:56:04.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-21"}]: dispatch 2024-03-11T20:56:04.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-21"}]: dispatch 2024-03-11T20:56:04.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2916558063' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:04.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:05.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:05 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:05.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:05 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-21"}]': finished 2024-03-11T20:56:05.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:05 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:05.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:05 smithi129 ceph-mon[30184]: osdmap e198: 8 total, 8 up, 8 in 2024-03-11T20:56:05.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-21", "mode": "writeback"}]: dispatch 2024-03-11T20:56:05.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:05 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-21", "mode": "writeback"}]: dispatch 2024-03-11T20:56:05.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:05 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:05.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:05 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-21"}]': finished 2024-03-11T20:56:05.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:05 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:05.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:05 smithi129 ceph-mon[42066]: osdmap e198: 8 total, 8 up, 8 in 2024-03-11T20:56:05.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-21", "mode": "writeback"}]: dispatch 2024-03-11T20:56:05.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:05 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-21", "mode": "writeback"}]: dispatch 2024-03-11T20:56:05.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:05 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:05.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:05 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-21"}]': finished 2024-03-11T20:56:05.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:05 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:05.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:05 smithi157 ceph-mon[33359]: osdmap e198: 8 total, 8 up, 8 in 2024-03-11T20:56:05.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-21", "mode": "writeback"}]: dispatch 2024-03-11T20:56:05.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:05 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-21", "mode": "writeback"}]: dispatch 2024-03-11T20:56:06.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:56:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:56:06.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[30184]: pgmap v251: 364 pgs: 13 creating+peering, 1 peering, 91 unknown, 259 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:56:06.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:06.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-21", "mode": "writeback"}]': finished 2024-03-11T20:56:06.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[30184]: osdmap e199: 8 total, 8 up, 8 in 2024-03-11T20:56:06.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2295066186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:06.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:06.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:06.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:06.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:06.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[42066]: pgmap v251: 364 pgs: 13 creating+peering, 1 peering, 91 unknown, 259 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:56:06.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:06.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-21", "mode": "writeback"}]': finished 2024-03-11T20:56:06.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[42066]: osdmap e199: 8 total, 8 up, 8 in 2024-03-11T20:56:06.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2295066186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:06.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:06.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:06.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:06.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:06 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:06 smithi157 ceph-mon[33359]: pgmap v251: 364 pgs: 13 creating+peering, 1 peering, 91 unknown, 259 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:56:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:06 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:06 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-21", "mode": "writeback"}]': finished 2024-03-11T20:56:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:06 smithi157 ceph-mon[33359]: osdmap e199: 8 total, 8 up, 8 in 2024-03-11T20:56:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2295066186' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:06.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:06 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:06.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:06.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:06 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2295066186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi129-82555-32"}]': finished 2024-03-11T20:56:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[42066]: osdmap e200: 8 total, 8 up, 8 in 2024-03-11T20:56:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[42066]: pgmap v254: 388 pgs: 32 creating+peering, 356 active+clean; 584 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-11T20:56:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21"}]: dispatch 2024-03-11T20:56:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21"}]: dispatch 2024-03-11T20:56:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2295066186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi129-82555-32"}]': finished 2024-03-11T20:56:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[30184]: osdmap e200: 8 total, 8 up, 8 in 2024-03-11T20:56:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[30184]: pgmap v254: 388 pgs: 32 creating+peering, 356 active+clean; 584 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-11T20:56:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21"}]: dispatch 2024-03-11T20:56:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21"}]: dispatch 2024-03-11T20:56:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2295066186' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi129-82567-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi129-82555-32"}]': finished 2024-03-11T20:56:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:07 smithi157 ceph-mon[33359]: osdmap e200: 8 total, 8 up, 8 in 2024-03-11T20:56:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2997090628' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi129-82555-32"}]: dispatch 2024-03-11T20:56:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:07 smithi157 ceph-mon[33359]: pgmap v254: 388 pgs: 32 creating+peering, 356 active+clean; 584 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-11T20:56:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21"}]: dispatch 2024-03-11T20:56:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21"}]: dispatch 2024-03-11T20:56:09.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:09.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi129-82555-32"}]': finished 2024-03-11T20:56:09.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21"}]': finished 2024-03-11T20:56:09.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[30184]: osdmap e201: 8 total, 8 up, 8 in 2024-03-11T20:56:09.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:09.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:09.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:09.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:09.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi129-82555-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:09.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi129-82555-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi129-82555-32"}]': finished 2024-03-11T20:56:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21"}]': finished 2024-03-11T20:56:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[42066]: osdmap e201: 8 total, 8 up, 8 in 2024-03-11T20:56:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:09.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:09.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi129-82555-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:09.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi129-82555-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:09.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:08 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:09.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi129-82555-32"}]': finished 2024-03-11T20:56:09.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-21"}]': finished 2024-03-11T20:56:09.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:08 smithi157 ceph-mon[33359]: osdmap e201: 8 total, 8 up, 8 in 2024-03-11T20:56:09.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:09.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:09.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:09.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:09.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi129-82555-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:09.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi129-82555-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:09 smithi129 ceph-mon[42066]: pgmap v256: 356 pgs: 356 active+clean; 584 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 765 B/s wr, 4 op/s 2024-03-11T20:56:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi129-82555-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:09 smithi129 ceph-mon[42066]: osdmap e202: 8 total, 8 up, 8 in 2024-03-11T20:56:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi129-82555-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi129-82555-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:10.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:09 smithi129 ceph-mon[30184]: pgmap v256: 356 pgs: 356 active+clean; 584 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 765 B/s wr, 4 op/s 2024-03-11T20:56:10.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi129-82555-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:10.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:09 smithi129 ceph-mon[30184]: osdmap e202: 8 total, 8 up, 8 in 2024-03-11T20:56:10.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi129-82555-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:10.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi129-82555-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:10.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:09 smithi157 ceph-mon[33359]: pgmap v256: 356 pgs: 356 active+clean; 584 KiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 765 B/s wr, 4 op/s 2024-03-11T20:56:10.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi129-82555-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:10.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:09 smithi157 ceph-mon[33359]: osdmap e202: 8 total, 8 up, 8 in 2024-03-11T20:56:10.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi129-82555-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:10.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi129-82555-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:11.228 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:10 smithi129 ceph-mon[30184]: osdmap e203: 8 total, 8 up, 8 in 2024-03-11T20:56:11.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:11.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:11.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:10 smithi129 ceph-mon[42066]: osdmap e203: 8 total, 8 up, 8 in 2024-03-11T20:56:11.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:11.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:11.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:10 smithi157 ceph-mon[33359]: osdmap e203: 8 total, 8 up, 8 in 2024-03-11T20:56:11.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:11.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:12.138 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:11 smithi157 ceph-mon[33359]: pgmap v259: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:12.138 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:11 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:12.138 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi129-82555-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi129-82555-33"}]': finished 2024-03-11T20:56:12.138 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:12.138 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:11 smithi157 ceph-mon[33359]: osdmap e204: 8 total, 8 up, 8 in 2024-03-11T20:56:12.138 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2615109581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:12.138 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:12.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:11 smithi129 ceph-mon[30184]: pgmap v259: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:12.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:11 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:12.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi129-82555-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi129-82555-33"}]': finished 2024-03-11T20:56:12.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:12.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:11 smithi129 ceph-mon[30184]: osdmap e204: 8 total, 8 up, 8 in 2024-03-11T20:56:12.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2615109581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:12.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:12.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:11 smithi129 ceph-mon[42066]: pgmap v259: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 564 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:12.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:11 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:12.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi129-82555-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi129-82555-33"}]': finished 2024-03-11T20:56:12.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:12.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:11 smithi129 ceph-mon[42066]: osdmap e204: 8 total, 8 up, 8 in 2024-03-11T20:56:12.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2615109581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:12.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:13.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:13.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:12 smithi129 ceph-mon[30184]: osdmap e205: 8 total, 8 up, 8 in 2024-03-11T20:56:13.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:13.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:13.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:13.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:12 smithi129 ceph-mon[42066]: osdmap e205: 8 total, 8 up, 8 in 2024-03-11T20:56:13.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:13.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:13.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:13.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:12 smithi157 ceph-mon[33359]: osdmap e205: 8 total, 8 up, 8 in 2024-03-11T20:56:13.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:13.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:13.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:56:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:56:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[42066]: pgmap v262: 332 pgs: 40 creating+peering, 292 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:56:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[42066]: osdmap e206: 8 total, 8 up, 8 in 2024-03-11T20:56:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-23"}]: dispatch 2024-03-11T20:56:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-23"}]: dispatch 2024-03-11T20:56:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4185713369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:14.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[30184]: pgmap v262: 332 pgs: 40 creating+peering, 292 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:56:14.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:14.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[30184]: osdmap e206: 8 total, 8 up, 8 in 2024-03-11T20:56:14.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-23"}]: dispatch 2024-03-11T20:56:14.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-23"}]: dispatch 2024-03-11T20:56:14.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4185713369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:14.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:14.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:14.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:14.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:13 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:14.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:13 smithi157 ceph-mon[33359]: pgmap v262: 332 pgs: 40 creating+peering, 292 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:56:14.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:14.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:13 smithi157 ceph-mon[33359]: osdmap e206: 8 total, 8 up, 8 in 2024-03-11T20:56:14.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-23"}]: dispatch 2024-03-11T20:56:14.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-23"}]: dispatch 2024-03-11T20:56:14.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4185713369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:14.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:14.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:14.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:14.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:13 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-23"}]': finished 2024-03-11T20:56:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi129-82555-33"}]': finished 2024-03-11T20:56:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[42066]: osdmap e207: 8 total, 8 up, 8 in 2024-03-11T20:56:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-23", "mode": "writeback"}]: dispatch 2024-03-11T20:56:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:15.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-23", "mode": "writeback"}]: dispatch 2024-03-11T20:56:15.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:15.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi129-82555-33"}]': finished 2024-03-11T20:56:15.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-23", "mode": "writeback"}]': finished 2024-03-11T20:56:15.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[42066]: osdmap e208: 8 total, 8 up, 8 in 2024-03-11T20:56:15.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-23"}]': finished 2024-03-11T20:56:15.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:15.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi129-82555-33"}]': finished 2024-03-11T20:56:15.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[30184]: osdmap e207: 8 total, 8 up, 8 in 2024-03-11T20:56:15.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:15.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-23", "mode": "writeback"}]: dispatch 2024-03-11T20:56:15.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:15.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-23", "mode": "writeback"}]: dispatch 2024-03-11T20:56:15.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:15.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi129-82555-33"}]': finished 2024-03-11T20:56:15.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-23", "mode": "writeback"}]': finished 2024-03-11T20:56:15.232 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:14 smithi129 ceph-mon[30184]: osdmap e208: 8 total, 8 up, 8 in 2024-03-11T20:56:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-23"}]': finished 2024-03-11T20:56:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi129-82555-33"}]': finished 2024-03-11T20:56:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:14 smithi157 ceph-mon[33359]: osdmap e207: 8 total, 8 up, 8 in 2024-03-11T20:56:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1558452546' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-23", "mode": "writeback"}]: dispatch 2024-03-11T20:56:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi129-82555-33"}]: dispatch 2024-03-11T20:56:15.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-23", "mode": "writeback"}]: dispatch 2024-03-11T20:56:15.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:14 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:15.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi129-82555-33"}]': finished 2024-03-11T20:56:15.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-23", "mode": "writeback"}]': finished 2024-03-11T20:56:15.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:14 smithi157 ceph-mon[33359]: osdmap e208: 8 total, 8 up, 8 in 2024-03-11T20:56:16.229 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:56:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:56:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:56:16.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[42066]: pgmap v265: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T20:56:16.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1828237780' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:16.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi129-82555-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi129-82555-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1828237780' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi129-82555-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[42066]: osdmap e209: 8 total, 8 up, 8 in 2024-03-11T20:56:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi129-82555-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi129-82555-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[30184]: pgmap v265: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T20:56:16.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1828237780' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi129-82555-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi129-82555-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1828237780' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi129-82555-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[30184]: osdmap e209: 8 total, 8 up, 8 in 2024-03-11T20:56:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi129-82555-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi129-82555-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:15 smithi157 ceph-mon[33359]: pgmap v265: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T20:56:16.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1828237780' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:16.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi129-82555-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi129-82555-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1828237780' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi129-82555-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:15 smithi157 ceph-mon[33359]: osdmap e209: 8 total, 8 up, 8 in 2024-03-11T20:56:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi129-82555-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi129-82555-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:17.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:17.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:17.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:17.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:18.106 INFO:tasks.workunit.client.0.smithi129.stdout: 0x558254a33000 2024-03-11T20:56:18.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a33110 2024-03-11T20:56:18.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a33240 2024-03-11T20:56:18.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a33bc0 2024-03-11T20:56:18.106 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a335c0 2024-03-11T20:56:18.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a33710 2024-03-11T20:56:18.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a33840 2024-03-11T20:56:18.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a33cd0 2024-03-11T20:56:18.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a34180 2024-03-11T20:56:18.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a342b0 2024-03-11T20:56:18.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a34890 2024-03-11T20:56:18.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a33de0 2024-03-11T20:56:18.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a33ef0 2024-03-11T20:56:18.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a34000 2024-03-11T20:56:18.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a349e0 2024-03-11T20:56:18.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a343e0 2024-03-11T20:56:18.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a32110 2024-03-11T20:56:18.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a32220 2024-03-11T20:56:18.107 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a32350 2024-03-11T20:56:18.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a34fe0 2024-03-11T20:56:18.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a34b30 2024-03-11T20:56:18.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a344f0 2024-03-11T20:56:18.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a34640 2024-03-11T20:56:18.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a34770 2024-03-11T20:56:18.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a355e0 2024-03-11T20:56:18.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a35130 2024-03-11T20:56:18.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a35280 2024-03-11T20:56:18.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a353b0 2024-03-11T20:56:18.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a356f0 2024-03-11T20:56:18.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a35ba0 2024-03-11T20:56:18.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a35cb0 2024-03-11T20:56:18.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a36270 2024-03-11T20:56:18.108 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a35800 2024-03-11T20:56:18.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a35910 2024-03-11T20:56:18.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a35a60 2024-03-11T20:56:18.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a36380 2024-03-11T20:56:18.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a35dc0 2024-03-11T20:56:18.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a35ef0 2024-03-11T20:56:18.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a36020 2024-03-11T20:56:18.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a36150 2024-03-11T20:56:18.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a36980 2024-03-11T20:56:18.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a364d0 2024-03-11T20:56:18.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a34c40 2024-03-11T20:56:18.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a34d50 2024-03-11T20:56:18.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a34ea0 2024-03-11T20:56:18.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a36f80 2024-03-11T20:56:18.109 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a36ad0 2024-03-11T20:56:18.110 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a365e0 2024-03-11T20:56:18.110 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a36710 2024-03-11T20:56:18.110 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a36820 2024-03-11T20:56:18.110 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a37580 2024-03-11T20:56:18.110 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a370d0 2024-03-11T20:56:18.110 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a36be0 2024-03-11T20:56:18.110 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a36d10 2024-03-11T20:56:18.110 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a36e40 2024-03-11T20:56:18.110 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a37b80 2024-03-11T20:56:18.110 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a376d0 2024-03-11T20:56:18.111 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a37cb0 2024-03-11T20:56:18.111 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a37de0 2024-03-11T20:56:18.111 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a383c0 2024-03-11T20:56:18.111 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a377e0 2024-03-11T20:56:18.111 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a37910 2024-03-11T20:56:18.111 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a37a40 2024-03-11T20:56:18.111 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a384d0 2024-03-11T20:56:18.111 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a37f10 2024-03-11T20:56:18.111 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a38040 2024-03-11T20:56:18.111 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a38150 2024-03-11T20:56:18.111 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a38280 2024-03-11T20:56:18.111 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a38ad0 2024-03-11T20:56:18.111 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a38620 2024-03-11T20:56:18.111 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a371e0 2024-03-11T20:56:18.112 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a372f0 2024-03-11T20:56:18.112 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a37440 2024-03-11T20:56:18.112 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a390d0 2024-03-11T20:56:18.112 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a38c20 2024-03-11T20:56:18.112 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a38d50 2024-03-11T20:56:18.112 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a38e60 2024-03-11T20:56:18.112 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a38fb0 2024-03-11T20:56:18.112 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a396d0 2024-03-11T20:56:18.112 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a39220 2024-03-11T20:56:18.113 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a39350 2024-03-11T20:56:18.113 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a39460 2024-03-11T20:56:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[30184]: pgmap v268: 388 pgs: 388 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-11T20:56:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23"}]: dispatch 2024-03-11T20:56:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[30184]: osdmap e210: 8 total, 8 up, 8 in 2024-03-11T20:56:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3026489798' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23"}]: dispatch 2024-03-11T20:56:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi129-82555-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi129-82555-34"}]': finished 2024-03-11T20:56:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23"}]': finished 2024-03-11T20:56:18.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:18.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[30184]: osdmap e211: 8 total, 8 up, 8 in 2024-03-11T20:56:18.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[42066]: pgmap v268: 388 pgs: 388 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-11T20:56:18.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23"}]: dispatch 2024-03-11T20:56:18.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[42066]: osdmap e210: 8 total, 8 up, 8 in 2024-03-11T20:56:18.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3026489798' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:18.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23"}]: dispatch 2024-03-11T20:56:18.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:18.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:18.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi129-82555-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi129-82555-34"}]': finished 2024-03-11T20:56:18.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23"}]': finished 2024-03-11T20:56:18.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:18.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:18 smithi129 ceph-mon[42066]: osdmap e211: 8 total, 8 up, 8 in 2024-03-11T20:56:18.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:18 smithi157 ceph-mon[33359]: pgmap v268: 388 pgs: 388 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-11T20:56:18.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23"}]: dispatch 2024-03-11T20:56:18.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:18 smithi157 ceph-mon[33359]: osdmap e210: 8 total, 8 up, 8 in 2024-03-11T20:56:18.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3026489798' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23"}]: dispatch 2024-03-11T20:56:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:18 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi129-82555-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi129-82555-34"}]': finished 2024-03-11T20:56:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-23"}]': finished 2024-03-11T20:56:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:18 smithi157 ceph-mon[33359]: osdmap e211: 8 total, 8 up, 8 in 2024-03-11T20:56:19.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:19.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:19 smithi129 ceph-mon[30184]: osdmap e212: 8 total, 8 up, 8 in 2024-03-11T20:56:19.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1822511455' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:19.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:19.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:19.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:19 smithi129 ceph-mon[42066]: osdmap e212: 8 total, 8 up, 8 in 2024-03-11T20:56:19.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1822511455' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:19.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:19.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:19.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:19 smithi157 ceph-mon[33359]: osdmap e212: 8 total, 8 up, 8 in 2024-03-11T20:56:19.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1822511455' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:19.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:20.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:20 smithi129 ceph-mon[30184]: pgmap v271: 428 pgs: 40 unknown, 388 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-11T20:56:20.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:20 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:20.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:20.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:20 smithi129 ceph-mon[30184]: osdmap e213: 8 total, 8 up, 8 in 2024-03-11T20:56:20.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:20.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:20.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:20.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:20.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:20 smithi129 ceph-mon[42066]: pgmap v271: 428 pgs: 40 unknown, 388 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-11T20:56:20.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:20 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:20.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:20.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:20 smithi129 ceph-mon[42066]: osdmap e213: 8 total, 8 up, 8 in 2024-03-11T20:56:20.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:20.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:20.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:20.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:20.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:20 smithi157 ceph-mon[33359]: pgmap v271: 428 pgs: 40 unknown, 388 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-11T20:56:20.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:20 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:20.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi129-82567-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:20.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:20 smithi157 ceph-mon[33359]: osdmap e213: 8 total, 8 up, 8 in 2024-03-11T20:56:20.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:20.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:20.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:20.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:21.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi129-82555-34"}]': finished 2024-03-11T20:56:21.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:21.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:21 smithi129 ceph-mon[42066]: osdmap e214: 8 total, 8 up, 8 in 2024-03-11T20:56:21.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:21.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:21.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi129-82555-34"}]': finished 2024-03-11T20:56:21.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:21.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:21 smithi129 ceph-mon[30184]: osdmap e214: 8 total, 8 up, 8 in 2024-03-11T20:56:21.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:21.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:21.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi129-82555-34"}]': finished 2024-03-11T20:56:21.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:21.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:21 smithi157 ceph-mon[33359]: osdmap e214: 8 total, 8 up, 8 in 2024-03-11T20:56:21.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/254934240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:21.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi129-82555-34"}]: dispatch 2024-03-11T20:56:22.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:22 smithi129 ceph-mon[42066]: pgmap v275: 420 pgs: 64 unknown, 356 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:22.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi129-82555-34"}]': finished 2024-03-11T20:56:22.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:22 smithi129 ceph-mon[42066]: osdmap e215: 8 total, 8 up, 8 in 2024-03-11T20:56:22.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi129-82555-35"}]: dispatch 2024-03-11T20:56:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi129-82555-35"}]: dispatch 2024-03-11T20:56:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi129-82555-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:22.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:22 smithi129 ceph-mon[30184]: pgmap v275: 420 pgs: 64 unknown, 356 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:22.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi129-82555-34"}]': finished 2024-03-11T20:56:22.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:22 smithi129 ceph-mon[30184]: osdmap e215: 8 total, 8 up, 8 in 2024-03-11T20:56:22.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:22.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:22.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi129-82555-35"}]: dispatch 2024-03-11T20:56:22.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi129-82555-35"}]: dispatch 2024-03-11T20:56:22.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi129-82555-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:22.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:22 smithi157 ceph-mon[33359]: pgmap v275: 420 pgs: 64 unknown, 356 active+clean; 584 KiB data, 549 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:22.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi129-82555-34"}]': finished 2024-03-11T20:56:22.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:22 smithi157 ceph-mon[33359]: osdmap e215: 8 total, 8 up, 8 in 2024-03-11T20:56:22.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:22.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:22.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi129-82555-35"}]: dispatch 2024-03-11T20:56:22.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi129-82555-35"}]: dispatch 2024-03-11T20:56:22.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi129-82555-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:23.691 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:56:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:56:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:23 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi129-82555-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:23 smithi129 ceph-mon[30184]: osdmap e216: 8 total, 8 up, 8 in 2024-03-11T20:56:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi129-82555-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi129-82555-35"}]: dispatch 2024-03-11T20:56:23.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-25"}]: dispatch 2024-03-11T20:56:23.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:23 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-25"}]: dispatch 2024-03-11T20:56:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:23 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi129-82555-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:23 smithi129 ceph-mon[42066]: osdmap e216: 8 total, 8 up, 8 in 2024-03-11T20:56:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi129-82555-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi129-82555-35"}]: dispatch 2024-03-11T20:56:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-25"}]: dispatch 2024-03-11T20:56:23.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:23 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-25"}]: dispatch 2024-03-11T20:56:23.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:23 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:23.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi129-82555-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:23.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:23 smithi157 ceph-mon[33359]: osdmap e216: 8 total, 8 up, 8 in 2024-03-11T20:56:23.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi129-82555-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi129-82555-35"}]: dispatch 2024-03-11T20:56:23.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-25"}]: dispatch 2024-03-11T20:56:23.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:23 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-25"}]: dispatch 2024-03-11T20:56:24.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:24 smithi129 ceph-mon[30184]: pgmap v278: 356 pgs: 356 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 519 B/s wr, 2 op/s 2024-03-11T20:56:24.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:24 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:24.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-25"}]': finished 2024-03-11T20:56:24.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-25", "mode": "writeback"}]: dispatch 2024-03-11T20:56:24.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:24 smithi129 ceph-mon[30184]: osdmap e217: 8 total, 8 up, 8 in 2024-03-11T20:56:24.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-25", "mode": "writeback"}]: dispatch 2024-03-11T20:56:24.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:24 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:24 smithi129 ceph-mon[42066]: pgmap v278: 356 pgs: 356 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 519 B/s wr, 2 op/s 2024-03-11T20:56:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:24 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-25"}]': finished 2024-03-11T20:56:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-25", "mode": "writeback"}]: dispatch 2024-03-11T20:56:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:24 smithi129 ceph-mon[42066]: osdmap e217: 8 total, 8 up, 8 in 2024-03-11T20:56:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-25", "mode": "writeback"}]: dispatch 2024-03-11T20:56:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:24 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:24.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:24 smithi157 ceph-mon[33359]: pgmap v278: 356 pgs: 356 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 519 B/s wr, 2 op/s 2024-03-11T20:56:24.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:24 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:24.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-25"}]': finished 2024-03-11T20:56:24.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-25", "mode": "writeback"}]: dispatch 2024-03-11T20:56:24.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:24 smithi157 ceph-mon[33359]: osdmap e217: 8 total, 8 up, 8 in 2024-03-11T20:56:24.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-25", "mode": "writeback"}]: dispatch 2024-03-11T20:56:24.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:24 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:25.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:25 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:25.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi129-82555-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi129-82555-35"}]': finished 2024-03-11T20:56:25.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-25", "mode": "writeback"}]': finished 2024-03-11T20:56:25.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:25 smithi129 ceph-mon[30184]: osdmap e218: 8 total, 8 up, 8 in 2024-03-11T20:56:25.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:25.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:25.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:25 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:25.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi129-82555-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi129-82555-35"}]': finished 2024-03-11T20:56:25.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-25", "mode": "writeback"}]': finished 2024-03-11T20:56:25.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:25 smithi129 ceph-mon[42066]: osdmap e218: 8 total, 8 up, 8 in 2024-03-11T20:56:25.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:25.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:25.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:25 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:25.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi129-82555-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi129-82555-35"}]': finished 2024-03-11T20:56:25.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-25", "mode": "writeback"}]': finished 2024-03-11T20:56:25.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:25 smithi157 ceph-mon[33359]: osdmap e218: 8 total, 8 up, 8 in 2024-03-11T20:56:25.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:25.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:26.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:56:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:56:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:56:26.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:26 smithi129 ceph-mon[30184]: pgmap v281: 300 pgs: 8 unknown, 292 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:56:26.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:26 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:26.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:26 smithi129 ceph-mon[30184]: osdmap e219: 8 total, 8 up, 8 in 2024-03-11T20:56:26.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:26 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25"}]: dispatch 2024-03-11T20:56:26.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:26 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3913801150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi129-82567-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:26.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:26 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25"}]: dispatch 2024-03-11T20:56:26.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:26 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi129-82567-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:26.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:26 smithi129 ceph-mon[42066]: pgmap v281: 300 pgs: 8 unknown, 292 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:56:26.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:26 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:26.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:26 smithi129 ceph-mon[42066]: osdmap e219: 8 total, 8 up, 8 in 2024-03-11T20:56:26.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:26 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25"}]: dispatch 2024-03-11T20:56:26.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:26 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3913801150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi129-82567-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:26.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:26 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25"}]: dispatch 2024-03-11T20:56:26.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:26 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi129-82567-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:26.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:26 smithi157 ceph-mon[33359]: pgmap v281: 300 pgs: 8 unknown, 292 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:56:26.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:26 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:26.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:26 smithi157 ceph-mon[33359]: osdmap e219: 8 total, 8 up, 8 in 2024-03-11T20:56:26.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:26 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25"}]: dispatch 2024-03-11T20:56:26.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:26 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3913801150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi129-82567-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:26.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:26 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25"}]: dispatch 2024-03-11T20:56:26.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:26 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi129-82567-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:28 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25"}]': finished 2024-03-11T20:56:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi129-82567-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:28 smithi129 ceph-mon[30184]: osdmap e220: 8 total, 8 up, 8 in 2024-03-11T20:56:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi129-82555-35"}]: dispatch 2024-03-11T20:56:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi129-82555-35"}]': finished 2024-03-11T20:56:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:28 smithi129 ceph-mon[30184]: osdmap e221: 8 total, 8 up, 8 in 2024-03-11T20:56:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi129-82555-35"}]: dispatch 2024-03-11T20:56:28.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:28 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:28.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25"}]': finished 2024-03-11T20:56:28.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi129-82567-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:28.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:28 smithi129 ceph-mon[42066]: osdmap e220: 8 total, 8 up, 8 in 2024-03-11T20:56:28.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi129-82555-35"}]: dispatch 2024-03-11T20:56:28.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi129-82555-35"}]': finished 2024-03-11T20:56:28.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:28 smithi129 ceph-mon[42066]: osdmap e221: 8 total, 8 up, 8 in 2024-03-11T20:56:28.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi129-82555-35"}]: dispatch 2024-03-11T20:56:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:28 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-25"}]': finished 2024-03-11T20:56:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi129-82567-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:28 smithi157 ceph-mon[33359]: osdmap e220: 8 total, 8 up, 8 in 2024-03-11T20:56:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi129-82555-35"}]: dispatch 2024-03-11T20:56:28.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi129-82555-35"}]': finished 2024-03-11T20:56:28.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:28 smithi157 ceph-mon[33359]: osdmap e221: 8 total, 8 up, 8 in 2024-03-11T20:56:28.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi129-82555-35"}]: dispatch 2024-03-11T20:56:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[30184]: pgmap v284: 324 pgs: 324 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T20:56:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi129-82555-35"}]': finished 2024-03-11T20:56:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[30184]: osdmap e222: 8 total, 8 up, 8 in 2024-03-11T20:56:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2082106717' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi129-82567-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi129-82567-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:29.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:29.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:29.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:29.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:29.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi129-82555-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:29.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi129-82555-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[42066]: pgmap v284: 324 pgs: 324 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T20:56:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi129-82555-35"}]': finished 2024-03-11T20:56:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[42066]: osdmap e222: 8 total, 8 up, 8 in 2024-03-11T20:56:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2082106717' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi129-82567-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:29.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:29.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi129-82567-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:29.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:29.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:29.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:29.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:29.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi129-82555-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:29.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi129-82555-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:29 smithi157 ceph-mon[33359]: pgmap v284: 324 pgs: 324 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T20:56:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/329711832' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi129-82555-35"}]': finished 2024-03-11T20:56:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:29 smithi157 ceph-mon[33359]: osdmap e222: 8 total, 8 up, 8 in 2024-03-11T20:56:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2082106717' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi129-82567-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:29.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi129-82567-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:29.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:29.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:29.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:29.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:29.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi129-82555-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:29.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi129-82555-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:30 smithi129 ceph-mon[30184]: pgmap v287: 324 pgs: 64 unknown, 260 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:56:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:30 smithi129 ceph-mon[30184]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:30 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:30 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi129-82567-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:30 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi129-82555-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:30 smithi129 ceph-mon[30184]: osdmap e223: 8 total, 8 up, 8 in 2024-03-11T20:56:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi129-82555-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:30 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi129-82555-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:30.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:30 smithi129 ceph-mon[42066]: pgmap v287: 324 pgs: 64 unknown, 260 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:56:30.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:30 smithi129 ceph-mon[42066]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:30.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:30 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:30.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:30 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi129-82567-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:30.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:30 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi129-82555-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:30.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:30 smithi129 ceph-mon[42066]: osdmap e223: 8 total, 8 up, 8 in 2024-03-11T20:56:30.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi129-82555-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:30.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:30 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi129-82555-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:30.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:30 smithi157 ceph-mon[33359]: pgmap v287: 324 pgs: 64 unknown, 260 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:56:30.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:30 smithi157 ceph-mon[33359]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:30.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:30 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:30.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:30 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi129-82567-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:30.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:30 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi129-82555-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:30.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:30 smithi157 ceph-mon[33359]: osdmap e223: 8 total, 8 up, 8 in 2024-03-11T20:56:30.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi129-82555-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:30.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:30 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi129-82555-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:31 smithi157 ceph-mon[33359]: osdmap e224: 8 total, 8 up, 8 in 2024-03-11T20:56:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:32.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[30184]: osdmap e224: 8 total, 8 up, 8 in 2024-03-11T20:56:32.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:32.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:32.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:32.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:32.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:32.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:32.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:32.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:32.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[42066]: osdmap e224: 8 total, 8 up, 8 in 2024-03-11T20:56:32.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:32.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:32.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:32.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:32.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:32.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:32.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:32.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[30184]: pgmap v290: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi129-82555-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi129-82555-36"}]': finished 2024-03-11T20:56:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[30184]: osdmap e225: 8 total, 8 up, 8 in 2024-03-11T20:56:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-27"}]: dispatch 2024-03-11T20:56:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-27"}]: dispatch 2024-03-11T20:56:33.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[42066]: pgmap v290: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:33.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi129-82555-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi129-82555-36"}]': finished 2024-03-11T20:56:33.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:33.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:33.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[42066]: osdmap e225: 8 total, 8 up, 8 in 2024-03-11T20:56:33.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:33.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:33.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-27"}]: dispatch 2024-03-11T20:56:33.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:32 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-27"}]: dispatch 2024-03-11T20:56:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:32 smithi157 ceph-mon[33359]: pgmap v290: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:32 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi129-82555-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi129-82555-36"}]': finished 2024-03-11T20:56:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:32 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:32 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:32 smithi157 ceph-mon[33359]: osdmap e225: 8 total, 8 up, 8 in 2024-03-11T20:56:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:32 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:32 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:33.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:32 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-27"}]: dispatch 2024-03-11T20:56:33.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:32 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-27"}]: dispatch 2024-03-11T20:56:33.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:56:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:56:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-27"}]': finished 2024-03-11T20:56:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-27", "mode": "writeback"}]: dispatch 2024-03-11T20:56:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:33 smithi157 ceph-mon[33359]: pgmap v293: 300 pgs: 8 creating+peering, 292 active+clean; 584 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T20:56:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:33 smithi157 ceph-mon[33359]: osdmap e226: 8 total, 8 up, 8 in 2024-03-11T20:56:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-27", "mode": "writeback"}]: dispatch 2024-03-11T20:56:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:33 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:33 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:33 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-46"}]': finished 2024-03-11T20:56:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-27", "mode": "writeback"}]': finished 2024-03-11T20:56:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:33 smithi157 ceph-mon[33359]: osdmap e227: 8 total, 8 up, 8 in 2024-03-11T20:56:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:34.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-27"}]': finished 2024-03-11T20:56:34.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-27", "mode": "writeback"}]: dispatch 2024-03-11T20:56:34.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[30184]: pgmap v293: 300 pgs: 8 creating+peering, 292 active+clean; 584 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T20:56:34.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[30184]: osdmap e226: 8 total, 8 up, 8 in 2024-03-11T20:56:34.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-27", "mode": "writeback"}]: dispatch 2024-03-11T20:56:34.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:34.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:34.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:34.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-46"}]': finished 2024-03-11T20:56:34.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-27", "mode": "writeback"}]': finished 2024-03-11T20:56:34.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[30184]: osdmap e227: 8 total, 8 up, 8 in 2024-03-11T20:56:34.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:34.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:34.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-27"}]': finished 2024-03-11T20:56:34.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-27", "mode": "writeback"}]: dispatch 2024-03-11T20:56:34.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[42066]: pgmap v293: 300 pgs: 8 creating+peering, 292 active+clean; 584 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T20:56:34.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[42066]: osdmap e226: 8 total, 8 up, 8 in 2024-03-11T20:56:34.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-27", "mode": "writeback"}]: dispatch 2024-03-11T20:56:34.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:34.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:34.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:34.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-46"}]': finished 2024-03-11T20:56:34.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-27", "mode": "writeback"}]': finished 2024-03-11T20:56:34.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[42066]: osdmap e227: 8 total, 8 up, 8 in 2024-03-11T20:56:34.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:34.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:36.106 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:35 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi129-82555-36"}]': finished 2024-03-11T20:56:36.106 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:35 smithi129 ceph-mon[30184]: osdmap e228: 8 total, 8 up, 8 in 2024-03-11T20:56:36.106 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:36.106 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:35 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:36.106 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:36.106 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:35 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:36.106 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:35 smithi129 ceph-mon[30184]: pgmap v296: 300 pgs: 8 unknown, 292 active+clean; 584 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T20:56:36.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:35 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi129-82555-36"}]': finished 2024-03-11T20:56:36.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:35 smithi129 ceph-mon[42066]: osdmap e228: 8 total, 8 up, 8 in 2024-03-11T20:56:36.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:36.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:35 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:36.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:36.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:35 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:36.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:35 smithi129 ceph-mon[42066]: pgmap v296: 300 pgs: 8 unknown, 292 active+clean; 584 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T20:56:36.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:35 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi129-82555-36"}]': finished 2024-03-11T20:56:36.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:35 smithi157 ceph-mon[33359]: osdmap e228: 8 total, 8 up, 8 in 2024-03-11T20:56:36.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1717879607' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:36.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:35 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi129-82555-36"}]: dispatch 2024-03-11T20:56:36.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:36.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:35 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:36.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:35 smithi157 ceph-mon[33359]: pgmap v296: 300 pgs: 8 unknown, 292 active+clean; 584 KiB data, 547 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T20:56:36.478 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:56:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:56:37.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi129-82555-36"}]': finished 2024-03-11T20:56:37.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:37.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[30184]: osdmap e229: 8 total, 8 up, 8 in 2024-03-11T20:56:37.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27"}]: dispatch 2024-03-11T20:56:37.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:37.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27"}]: dispatch 2024-03-11T20:56:37.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:37.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi129-82555-37"}]: dispatch 2024-03-11T20:56:37.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi129-82555-37"}]: dispatch 2024-03-11T20:56:37.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi129-82555-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:37.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi129-82555-36"}]': finished 2024-03-11T20:56:37.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:37.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[42066]: osdmap e229: 8 total, 8 up, 8 in 2024-03-11T20:56:37.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27"}]: dispatch 2024-03-11T20:56:37.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:37.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27"}]: dispatch 2024-03-11T20:56:37.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:37.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi129-82555-37"}]: dispatch 2024-03-11T20:56:37.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi129-82555-37"}]: dispatch 2024-03-11T20:56:37.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi129-82555-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:37.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi129-82555-36"}]': finished 2024-03-11T20:56:37.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:37.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:36 smithi157 ceph-mon[33359]: osdmap e229: 8 total, 8 up, 8 in 2024-03-11T20:56:37.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27"}]: dispatch 2024-03-11T20:56:37.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:37.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27"}]: dispatch 2024-03-11T20:56:37.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:37.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi129-82555-37"}]: dispatch 2024-03-11T20:56:37.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi129-82555-37"}]: dispatch 2024-03-11T20:56:37.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi129-82555-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:38.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:38.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27"}]': finished 2024-03-11T20:56:38.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-46"}]': finished 2024-03-11T20:56:38.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi129-82555-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:38.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[30184]: osdmap e230: 8 total, 8 up, 8 in 2024-03-11T20:56:38.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:38.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi129-82555-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi129-82555-37"}]: dispatch 2024-03-11T20:56:38.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:38.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[30184]: pgmap v299: 292 pgs: 292 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T20:56:38.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:38.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27"}]': finished 2024-03-11T20:56:38.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-46"}]': finished 2024-03-11T20:56:38.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi129-82555-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:38.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[42066]: osdmap e230: 8 total, 8 up, 8 in 2024-03-11T20:56:38.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:38.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi129-82555-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi129-82555-37"}]: dispatch 2024-03-11T20:56:38.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:38.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:37 smithi129 ceph-mon[42066]: pgmap v299: 292 pgs: 292 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T20:56:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:37 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-27"}]': finished 2024-03-11T20:56:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-46"}]': finished 2024-03-11T20:56:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi129-82555-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:37 smithi157 ceph-mon[33359]: osdmap e230: 8 total, 8 up, 8 in 2024-03-11T20:56:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1153505535' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi129-82555-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi129-82555-37"}]: dispatch 2024-03-11T20:56:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-46"}]: dispatch 2024-03-11T20:56:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:37 smithi157 ceph-mon[33359]: pgmap v299: 292 pgs: 292 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T20:56:39.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:39.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-46"}]': finished 2024-03-11T20:56:39.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:39.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[30184]: osdmap e231: 8 total, 8 up, 8 in 2024-03-11T20:56:39.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:39.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:39.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:39.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:39.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:39.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:39.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-46"}]': finished 2024-03-11T20:56:39.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:39.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[42066]: osdmap e231: 8 total, 8 up, 8 in 2024-03-11T20:56:39.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:39.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:39.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:39.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:39.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:39.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:38 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:39.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-46"}]': finished 2024-03-11T20:56:39.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:39.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:38 smithi157 ceph-mon[33359]: osdmap e231: 8 total, 8 up, 8 in 2024-03-11T20:56:39.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:39.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:39.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:39.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:39.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:40.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi129-82555-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi129-82555-37"}]': finished 2024-03-11T20:56:40.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:40.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:39 smithi129 ceph-mon[30184]: osdmap e232: 8 total, 8 up, 8 in 2024-03-11T20:56:40.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:40.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:40.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:40.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:40.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:39 smithi129 ceph-mon[30184]: pgmap v302: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:56:40.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi129-82555-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi129-82555-37"}]': finished 2024-03-11T20:56:40.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:40.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:39 smithi129 ceph-mon[42066]: osdmap e232: 8 total, 8 up, 8 in 2024-03-11T20:56:40.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:40.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:40.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:40.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:40.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:39 smithi129 ceph-mon[42066]: pgmap v302: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:56:40.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi129-82555-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi129-82555-37"}]': finished 2024-03-11T20:56:40.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi129-82567-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:40.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:39 smithi157 ceph-mon[33359]: osdmap e232: 8 total, 8 up, 8 in 2024-03-11T20:56:40.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:40.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:40.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:40.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:40.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:39 smithi157 ceph-mon[33359]: pgmap v302: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:56:41.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:41.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:40 smithi129 ceph-mon[42066]: osdmap e233: 8 total, 8 up, 8 in 2024-03-11T20:56:41.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:41.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:40 smithi129 ceph-mon[30184]: osdmap e233: 8 total, 8 up, 8 in 2024-03-11T20:56:41.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:41.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:40 smithi157 ceph-mon[33359]: osdmap e233: 8 total, 8 up, 8 in 2024-03-11T20:56:42.136 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:41 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-47"}]': finished 2024-03-11T20:56:42.137 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:41 smithi157 ceph-mon[33359]: osdmap e234: 8 total, 8 up, 8 in 2024-03-11T20:56:42.137 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:41 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi129-82555-37"}]: dispatch 2024-03-11T20:56:42.137 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:41 smithi157 ceph-mon[33359]: pgmap v305: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:42.137 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:56:42.137 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:56:42.137 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:41 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:56:42.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:41 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-47"}]': finished 2024-03-11T20:56:42.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:41 smithi129 ceph-mon[42066]: osdmap e234: 8 total, 8 up, 8 in 2024-03-11T20:56:42.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:41 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi129-82555-37"}]: dispatch 2024-03-11T20:56:42.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:41 smithi129 ceph-mon[42066]: pgmap v305: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:42.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:56:42.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:56:42.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:41 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:56:42.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:41 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi129-82567-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi129-82567-47"}]': finished 2024-03-11T20:56:42.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:41 smithi129 ceph-mon[30184]: osdmap e234: 8 total, 8 up, 8 in 2024-03-11T20:56:42.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:41 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi129-82555-37"}]: dispatch 2024-03-11T20:56:42.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:41 smithi129 ceph-mon[30184]: pgmap v305: 300 pgs: 40 unknown, 260 active+clean; 584 KiB data, 548 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:42.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:56:42.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:56:42.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:41 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:56:43.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi129-82555-37"}]': finished 2024-03-11T20:56:43.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:42 smithi129 ceph-mon[30184]: osdmap e235: 8 total, 8 up, 8 in 2024-03-11T20:56:43.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi129-82555-37"}]: dispatch 2024-03-11T20:56:43.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:43.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:42 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:43.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:42 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:56:43.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:42 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:56:43.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:42 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:56:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi129-82555-37"}]': finished 2024-03-11T20:56:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:42 smithi129 ceph-mon[42066]: osdmap e235: 8 total, 8 up, 8 in 2024-03-11T20:56:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi129-82555-37"}]: dispatch 2024-03-11T20:56:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:42 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:42 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:56:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:42 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:56:43.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:42 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:56:43.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi129-82555-37"}]': finished 2024-03-11T20:56:43.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:42 smithi157 ceph-mon[33359]: osdmap e235: 8 total, 8 up, 8 in 2024-03-11T20:56:43.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi129-82555-37"}]: dispatch 2024-03-11T20:56:43.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:43.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:42 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:43.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:42 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:56:43.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:42 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:56:43.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:42 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:56:43.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:56:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:56:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi129-82555-37"}]': finished 2024-03-11T20:56:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[30184]: osdmap e236: 8 total, 8 up, 8 in 2024-03-11T20:56:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-29"}]: dispatch 2024-03-11T20:56:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:44.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-29"}]: dispatch 2024-03-11T20:56:44.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:44.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[30184]: pgmap v308: 292 pgs: 292 active+clean; 4.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-11T20:56:44.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi129-82555-38"}]: dispatch 2024-03-11T20:56:44.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi129-82555-38"}]: dispatch 2024-03-11T20:56:44.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi129-82555-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:44.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:44.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi129-82555-37"}]': finished 2024-03-11T20:56:44.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:44.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[42066]: osdmap e236: 8 total, 8 up, 8 in 2024-03-11T20:56:44.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-29"}]: dispatch 2024-03-11T20:56:44.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:44.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-29"}]: dispatch 2024-03-11T20:56:44.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:44.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[42066]: pgmap v308: 292 pgs: 292 active+clean; 4.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-11T20:56:44.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi129-82555-38"}]: dispatch 2024-03-11T20:56:44.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi129-82555-38"}]: dispatch 2024-03-11T20:56:44.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi129-82555-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:44.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:43 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1854445957' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi129-82555-37"}]': finished 2024-03-11T20:56:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:43 smithi157 ceph-mon[33359]: osdmap e236: 8 total, 8 up, 8 in 2024-03-11T20:56:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-29"}]: dispatch 2024-03-11T20:56:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-29"}]: dispatch 2024-03-11T20:56:44.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:44.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:43 smithi157 ceph-mon[33359]: pgmap v308: 292 pgs: 292 active+clean; 4.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-11T20:56:44.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi129-82555-38"}]: dispatch 2024-03-11T20:56:44.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi129-82555-38"}]: dispatch 2024-03-11T20:56:44.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi129-82555-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:44.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:43 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:45.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:45.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-29"}]': finished 2024-03-11T20:56:45.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-47"}]': finished 2024-03-11T20:56:45.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi129-82555-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:45.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[30184]: osdmap e237: 8 total, 8 up, 8 in 2024-03-11T20:56:45.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:45.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-29", "mode": "writeback"}]: dispatch 2024-03-11T20:56:45.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi129-82555-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi129-82555-38"}]: dispatch 2024-03-11T20:56:45.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:45.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-29", "mode": "writeback"}]: dispatch 2024-03-11T20:56:45.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:45.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-47"}]': finished 2024-03-11T20:56:45.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:45.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-29"}]': finished 2024-03-11T20:56:45.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-47"}]': finished 2024-03-11T20:56:45.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi129-82555-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:45.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[42066]: osdmap e237: 8 total, 8 up, 8 in 2024-03-11T20:56:45.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:45.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-29", "mode": "writeback"}]: dispatch 2024-03-11T20:56:45.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi129-82555-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi129-82555-38"}]: dispatch 2024-03-11T20:56:45.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:45.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-29", "mode": "writeback"}]: dispatch 2024-03-11T20:56:45.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:45.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:44 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-47"}]': finished 2024-03-11T20:56:45.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:44 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:45.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:44 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-29"}]': finished 2024-03-11T20:56:45.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:44 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi129-82567-47"}]': finished 2024-03-11T20:56:45.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi129-82555-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:45.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:44 smithi157 ceph-mon[33359]: osdmap e237: 8 total, 8 up, 8 in 2024-03-11T20:56:45.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4145996723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:45.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-29", "mode": "writeback"}]: dispatch 2024-03-11T20:56:45.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi129-82555-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi129-82555-38"}]: dispatch 2024-03-11T20:56:45.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:44 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-47"}]: dispatch 2024-03-11T20:56:45.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:44 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-29", "mode": "writeback"}]: dispatch 2024-03-11T20:56:45.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:44 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:45.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:44 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi129-82567-47"}]': finished 2024-03-11T20:56:46.229 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:56:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:56:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:56:46.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-29", "mode": "writeback"}]': finished 2024-03-11T20:56:46.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[30184]: pgmap v311: 292 pgs: 292 active+clean; 4.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-11T20:56:46.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[30184]: osdmap e238: 8 total, 8 up, 8 in 2024-03-11T20:56:46.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi129-82567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:46.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi129-82567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:46.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi129-82555-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi129-82555-38"}]': finished 2024-03-11T20:56:46.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi129-82567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:46.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[30184]: osdmap e239: 8 total, 8 up, 8 in 2024-03-11T20:56:46.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi129-82567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi129-82567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-29", "mode": "writeback"}]': finished 2024-03-11T20:56:46.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[42066]: pgmap v311: 292 pgs: 292 active+clean; 4.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-11T20:56:46.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[42066]: osdmap e238: 8 total, 8 up, 8 in 2024-03-11T20:56:46.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi129-82567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:46.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi129-82567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:46.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi129-82555-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi129-82555-38"}]': finished 2024-03-11T20:56:46.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi129-82567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:46.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[42066]: osdmap e239: 8 total, 8 up, 8 in 2024-03-11T20:56:46.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi129-82567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:45 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi129-82567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:45 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-29", "mode": "writeback"}]': finished 2024-03-11T20:56:46.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:45 smithi157 ceph-mon[33359]: pgmap v311: 292 pgs: 292 active+clean; 4.6 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-11T20:56:46.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:45 smithi157 ceph-mon[33359]: osdmap e238: 8 total, 8 up, 8 in 2024-03-11T20:56:46.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:45 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:45 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi129-82567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:46.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:45 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi129-82567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:46.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi129-82555-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi129-82555-38"}]': finished 2024-03-11T20:56:46.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:45 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi129-82567-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:46.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:45 smithi157 ceph-mon[33359]: osdmap e239: 8 total, 8 up, 8 in 2024-03-11T20:56:46.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi129-82567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:46.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:45 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi129-82567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:47.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:47 smithi129 ceph-mon[30184]: osdmap e240: 8 total, 8 up, 8 in 2024-03-11T20:56:47.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:47 smithi129 ceph-mon[42066]: osdmap e240: 8 total, 8 up, 8 in 2024-03-11T20:56:47.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:47 smithi157 ceph-mon[33359]: osdmap e240: 8 total, 8 up, 8 in 2024-03-11T20:56:48.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:48 smithi129 ceph-mon[42066]: pgmap v314: 300 pgs: 1 creating+peering, 1 unknown, 298 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T20:56:48.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:48 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi129-82567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi129-82567-48"}]': finished 2024-03-11T20:56:48.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:48 smithi129 ceph-mon[42066]: osdmap e241: 8 total, 8 up, 8 in 2024-03-11T20:56:48.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi129-82555-38"}]: dispatch 2024-03-11T20:56:48.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:48.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:48 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:48.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:48 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:56:48.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:48 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:48.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:48 smithi129 ceph-mon[30184]: pgmap v314: 300 pgs: 1 creating+peering, 1 unknown, 298 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T20:56:48.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:48 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi129-82567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi129-82567-48"}]': finished 2024-03-11T20:56:48.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:48 smithi129 ceph-mon[30184]: osdmap e241: 8 total, 8 up, 8 in 2024-03-11T20:56:48.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi129-82555-38"}]: dispatch 2024-03-11T20:56:48.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:48.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:48 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:48.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:48 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:56:48.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:48 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:48.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:48 smithi157 ceph-mon[33359]: pgmap v314: 300 pgs: 1 creating+peering, 1 unknown, 298 active+clean; 4.6 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T20:56:48.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:48 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi129-82567-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi129-82567-48"}]': finished 2024-03-11T20:56:48.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:48 smithi157 ceph-mon[33359]: osdmap e241: 8 total, 8 up, 8 in 2024-03-11T20:56:48.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi129-82555-38"}]: dispatch 2024-03-11T20:56:48.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:48.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:48 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:48.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:48 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:56:48.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:48 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:56:49.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi129-82555-38"}]': finished 2024-03-11T20:56:49.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:49.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[30184]: osdmap e242: 8 total, 8 up, 8 in 2024-03-11T20:56:49.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi129-82555-38"}]: dispatch 2024-03-11T20:56:49.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29"}]: dispatch 2024-03-11T20:56:49.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29"}]: dispatch 2024-03-11T20:56:49.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:49.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:49.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi129-82555-38"}]': finished 2024-03-11T20:56:49.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29"}]': finished 2024-03-11T20:56:49.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[30184]: osdmap e243: 8 total, 8 up, 8 in 2024-03-11T20:56:49.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi129-82555-38"}]': finished 2024-03-11T20:56:49.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:49.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[42066]: osdmap e242: 8 total, 8 up, 8 in 2024-03-11T20:56:49.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi129-82555-38"}]: dispatch 2024-03-11T20:56:49.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29"}]: dispatch 2024-03-11T20:56:49.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29"}]: dispatch 2024-03-11T20:56:49.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:49.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:49.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi129-82555-38"}]': finished 2024-03-11T20:56:49.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29"}]': finished 2024-03-11T20:56:49.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:49 smithi129 ceph-mon[42066]: osdmap e243: 8 total, 8 up, 8 in 2024-03-11T20:56:49.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi129-82555-38"}]': finished 2024-03-11T20:56:49.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:49.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:49 smithi157 ceph-mon[33359]: osdmap e242: 8 total, 8 up, 8 in 2024-03-11T20:56:49.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi129-82555-38"}]: dispatch 2024-03-11T20:56:49.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29"}]: dispatch 2024-03-11T20:56:49.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29"}]: dispatch 2024-03-11T20:56:49.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:49 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:49.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:49 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:49.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3264994862' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi129-82555-38"}]': finished 2024-03-11T20:56:49.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-29"}]': finished 2024-03-11T20:56:49.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:49 smithi157 ceph-mon[33359]: osdmap e243: 8 total, 8 up, 8 in 2024-03-11T20:56:51.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[30184]: pgmap v317: 300 pgs: 8 unknown, 292 active+clean; 8.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-11T20:56:51.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:51.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:51.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi129-82555-39"}]: dispatch 2024-03-11T20:56:51.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi129-82555-39"}]: dispatch 2024-03-11T20:56:51.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi129-82555-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:51.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi129-82567-48"}]': finished 2024-03-11T20:56:51.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi129-82555-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:51.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[30184]: osdmap e244: 8 total, 8 up, 8 in 2024-03-11T20:56:51.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi129-82555-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi129-82555-39"}]: dispatch 2024-03-11T20:56:51.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:51.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[42066]: pgmap v317: 300 pgs: 8 unknown, 292 active+clean; 8.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-11T20:56:51.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:51.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:51.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi129-82555-39"}]: dispatch 2024-03-11T20:56:51.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi129-82555-39"}]: dispatch 2024-03-11T20:56:51.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi129-82555-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:51.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi129-82567-48"}]': finished 2024-03-11T20:56:51.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi129-82555-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:51.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[42066]: osdmap e244: 8 total, 8 up, 8 in 2024-03-11T20:56:51.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi129-82555-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi129-82555-39"}]: dispatch 2024-03-11T20:56:51.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:51.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:50 smithi157 ceph-mon[33359]: pgmap v317: 300 pgs: 8 unknown, 292 active+clean; 8.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-11T20:56:51.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:51.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:51.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi129-82555-39"}]: dispatch 2024-03-11T20:56:51.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi129-82555-39"}]: dispatch 2024-03-11T20:56:51.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi129-82555-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:51.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi129-82567-48"}]': finished 2024-03-11T20:56:51.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi129-82555-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:51.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:50 smithi157 ceph-mon[33359]: osdmap e244: 8 total, 8 up, 8 in 2024-03-11T20:56:51.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi129-82555-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi129-82555-39"}]: dispatch 2024-03-11T20:56:51.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1677000748' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:52.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:51 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:52.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:51 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi129-82567-48"}]': finished 2024-03-11T20:56:52.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:51 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:52.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:51 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi129-82567-48"}]': finished 2024-03-11T20:56:52.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:51 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi129-82567-48"}]: dispatch 2024-03-11T20:56:52.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:51 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi129-82567-48"}]': finished 2024-03-11T20:56:53.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: pgmap v320: 260 pgs: 260 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:53.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: osdmap e245: 8 total, 8 up, 8 in 2024-03-11T20:56:53.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:53.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:53.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi129-82567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:53.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi129-82567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:53.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi129-82555-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi129-82555-39"}]': finished 2024-03-11T20:56:53.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:53.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi129-82567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:53.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: osdmap e246: 8 total, 8 up, 8 in 2024-03-11T20:56:53.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi129-82567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:53.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi129-82567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:53.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: pgmap v320: 260 pgs: 260 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:53.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: osdmap e245: 8 total, 8 up, 8 in 2024-03-11T20:56:53.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:53.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:53.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi129-82567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:53.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi129-82567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:53.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi129-82555-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi129-82555-39"}]': finished 2024-03-11T20:56:53.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:53.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi129-82567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:53.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: osdmap e246: 8 total, 8 up, 8 in 2024-03-11T20:56:53.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi129-82567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:53.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi129-82567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:53.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: pgmap v320: 260 pgs: 260 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:53.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: osdmap e245: 8 total, 8 up, 8 in 2024-03-11T20:56:53.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:53.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:56:53.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi129-82567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:53.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi129-82567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:53.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi129-82555-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi129-82555-39"}]': finished 2024-03-11T20:56:53.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:56:53.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi129-82567-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:53.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: osdmap e246: 8 total, 8 up, 8 in 2024-03-11T20:56:53.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi129-82567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:53.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi129-82567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:53.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:56:53.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:56:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:56:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[30184]: pgmap v323: 300 pgs: 29 creating+peering, 11 unknown, 260 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-31"}]: dispatch 2024-03-11T20:56:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[30184]: osdmap e247: 8 total, 8 up, 8 in 2024-03-11T20:56:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-31"}]: dispatch 2024-03-11T20:56:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi129-82567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi129-82567-49"}]': finished 2024-03-11T20:56:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-31"}]': finished 2024-03-11T20:56:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[30184]: osdmap e248: 8 total, 8 up, 8 in 2024-03-11T20:56:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi129-82555-39"}]: dispatch 2024-03-11T20:56:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-31", "mode": "writeback"}]: dispatch 2024-03-11T20:56:54.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-31", "mode": "writeback"}]: dispatch 2024-03-11T20:56:54.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[42066]: pgmap v323: 300 pgs: 29 creating+peering, 11 unknown, 260 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:54.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:54.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:54.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-31"}]: dispatch 2024-03-11T20:56:54.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[42066]: osdmap e247: 8 total, 8 up, 8 in 2024-03-11T20:56:54.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-31"}]: dispatch 2024-03-11T20:56:54.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi129-82567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi129-82567-49"}]': finished 2024-03-11T20:56:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-31"}]': finished 2024-03-11T20:56:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[42066]: osdmap e248: 8 total, 8 up, 8 in 2024-03-11T20:56:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi129-82555-39"}]: dispatch 2024-03-11T20:56:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-31", "mode": "writeback"}]: dispatch 2024-03-11T20:56:54.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-31", "mode": "writeback"}]: dispatch 2024-03-11T20:56:54.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:54 smithi157 ceph-mon[33359]: pgmap v323: 300 pgs: 29 creating+peering, 11 unknown, 260 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:56:54.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:54 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:56:54.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:56:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-31"}]: dispatch 2024-03-11T20:56:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:54 smithi157 ceph-mon[33359]: osdmap e247: 8 total, 8 up, 8 in 2024-03-11T20:56:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-31"}]: dispatch 2024-03-11T20:56:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi129-82567-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi129-82567-49"}]': finished 2024-03-11T20:56:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-31"}]': finished 2024-03-11T20:56:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:54 smithi157 ceph-mon[33359]: osdmap e248: 8 total, 8 up, 8 in 2024-03-11T20:56:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi129-82555-39"}]: dispatch 2024-03-11T20:56:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-31", "mode": "writeback"}]: dispatch 2024-03-11T20:56:54.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-31", "mode": "writeback"}]: dispatch 2024-03-11T20:56:55.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:55 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:55.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi129-82555-39"}]': finished 2024-03-11T20:56:55.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-31", "mode": "writeback"}]': finished 2024-03-11T20:56:55.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:55 smithi129 ceph-mon[30184]: osdmap e249: 8 total, 8 up, 8 in 2024-03-11T20:56:55.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi129-82555-39"}]: dispatch 2024-03-11T20:56:55.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:55 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:55.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi129-82555-39"}]': finished 2024-03-11T20:56:55.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-31", "mode": "writeback"}]': finished 2024-03-11T20:56:55.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:55 smithi129 ceph-mon[42066]: osdmap e249: 8 total, 8 up, 8 in 2024-03-11T20:56:55.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi129-82555-39"}]: dispatch 2024-03-11T20:56:55.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:55 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:56:55.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi129-82555-39"}]': finished 2024-03-11T20:56:55.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-31", "mode": "writeback"}]': finished 2024-03-11T20:56:55.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:55 smithi157 ceph-mon[33359]: osdmap e249: 8 total, 8 up, 8 in 2024-03-11T20:56:55.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi129-82555-39"}]: dispatch 2024-03-11T20:56:56.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:56:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:56:56.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[30184]: pgmap v326: 300 pgs: 29 creating+peering, 11 unknown, 260 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-11T20:56:56.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:56.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:56.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi129-82555-39"}]': finished 2024-03-11T20:56:56.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:56.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[30184]: osdmap e250: 8 total, 8 up, 8 in 2024-03-11T20:56:56.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31"}]: dispatch 2024-03-11T20:56:56.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:56.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31"}]: dispatch 2024-03-11T20:56:56.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:56.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi129-82555-40"}]: dispatch 2024-03-11T20:56:56.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi129-82555-40"}]: dispatch 2024-03-11T20:56:56.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi129-82555-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:56.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[42066]: pgmap v326: 300 pgs: 29 creating+peering, 11 unknown, 260 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-11T20:56:56.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:56.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:56.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi129-82555-39"}]': finished 2024-03-11T20:56:56.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:56.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[42066]: osdmap e250: 8 total, 8 up, 8 in 2024-03-11T20:56:56.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31"}]: dispatch 2024-03-11T20:56:56.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:56.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31"}]: dispatch 2024-03-11T20:56:56.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:56.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi129-82555-40"}]: dispatch 2024-03-11T20:56:56.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi129-82555-40"}]: dispatch 2024-03-11T20:56:56.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi129-82555-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:56.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:56 smithi157 ceph-mon[33359]: pgmap v326: 300 pgs: 29 creating+peering, 11 unknown, 260 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-11T20:56:56.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:56.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:56 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:56:56.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1662816793' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi129-82555-39"}]': finished 2024-03-11T20:56:56.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:56 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:56:56.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:56 smithi157 ceph-mon[33359]: osdmap e250: 8 total, 8 up, 8 in 2024-03-11T20:56:56.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31"}]: dispatch 2024-03-11T20:56:56.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:56.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:56 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31"}]: dispatch 2024-03-11T20:56:56.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:56 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:56.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi129-82555-40"}]: dispatch 2024-03-11T20:56:56.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi129-82555-40"}]: dispatch 2024-03-11T20:56:56.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi129-82555-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:56:57.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:57 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:57.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31"}]': finished 2024-03-11T20:56:57.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi129-82567-49"}]': finished 2024-03-11T20:56:57.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi129-82555-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:57.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:57 smithi157 ceph-mon[33359]: osdmap e251: 8 total, 8 up, 8 in 2024-03-11T20:56:57.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi129-82555-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi129-82555-40"}]: dispatch 2024-03-11T20:56:57.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:57.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:58.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:57 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:58.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31"}]': finished 2024-03-11T20:56:58.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi129-82567-49"}]': finished 2024-03-11T20:56:58.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi129-82555-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:58.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:57 smithi129 ceph-mon[30184]: osdmap e251: 8 total, 8 up, 8 in 2024-03-11T20:56:58.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi129-82555-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi129-82555-40"}]: dispatch 2024-03-11T20:56:58.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:58.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:57 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:56:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-31"}]': finished 2024-03-11T20:56:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi129-82567-49"}]': finished 2024-03-11T20:56:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi129-82555-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:56:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:57 smithi129 ceph-mon[42066]: osdmap e251: 8 total, 8 up, 8 in 2024-03-11T20:56:58.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi129-82555-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi129-82555-40"}]: dispatch 2024-03-11T20:56:58.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3899316020' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:58.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi129-82567-49"}]: dispatch 2024-03-11T20:56:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:58 smithi129 ceph-mon[30184]: pgmap v329: 292 pgs: 4 creating+peering, 288 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:56:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:58 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:58 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi129-82567-49"}]': finished 2024-03-11T20:56:59.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:58 smithi129 ceph-mon[42066]: pgmap v329: 292 pgs: 4 creating+peering, 288 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:56:59.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:58 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:59.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:58 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi129-82567-49"}]': finished 2024-03-11T20:56:59.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:58 smithi157 ceph-mon[33359]: pgmap v329: 292 pgs: 4 creating+peering, 288 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:56:59.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:58 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:56:59.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:58 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi129-82567-49"}]': finished 2024-03-11T20:57:00.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[30184]: pgmap v332: 260 pgs: 260 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:57:00.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[30184]: osdmap e252: 8 total, 8 up, 8 in 2024-03-11T20:57:00.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi129-82567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:00.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi129-82567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:00.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi129-82555-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi129-82555-40"}]': finished 2024-03-11T20:57:00.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi129-82567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:00.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[30184]: osdmap e253: 8 total, 8 up, 8 in 2024-03-11T20:57:00.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi129-82567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi129-82567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:00.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:00.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[42066]: pgmap v332: 260 pgs: 260 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:57:00.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[42066]: osdmap e252: 8 total, 8 up, 8 in 2024-03-11T20:57:00.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi129-82567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:00.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi129-82567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:00.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi129-82555-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi129-82555-40"}]': finished 2024-03-11T20:57:00.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi129-82567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:00.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[42066]: osdmap e253: 8 total, 8 up, 8 in 2024-03-11T20:57:00.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi129-82567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi129-82567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:00.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:56:59 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:00.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:59 smithi157 ceph-mon[33359]: pgmap v332: 260 pgs: 260 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:57:00.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:59 smithi157 ceph-mon[33359]: osdmap e252: 8 total, 8 up, 8 in 2024-03-11T20:57:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:59 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:59 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi129-82567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:59 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi129-82567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi129-82555-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi129-82555-40"}]': finished 2024-03-11T20:57:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:59 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi129-82567-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:59 smithi157 ceph-mon[33359]: osdmap e253: 8 total, 8 up, 8 in 2024-03-11T20:57:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi129-82567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:59 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi129-82567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:00.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:56:59 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:01.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:01 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:01.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:01.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:01 smithi129 ceph-mon[30184]: osdmap e254: 8 total, 8 up, 8 in 2024-03-11T20:57:01.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:01 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:01.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:01 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:01.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:01.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:01 smithi129 ceph-mon[42066]: osdmap e254: 8 total, 8 up, 8 in 2024-03-11T20:57:01.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:01 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:01.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:01 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:01.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:01.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:01 smithi157 ceph-mon[33359]: osdmap e254: 8 total, 8 up, 8 in 2024-03-11T20:57:01.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:01 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:02.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:02 smithi129 ceph-mon[30184]: pgmap v335: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:02.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:02 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi129-82567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi129-82567-50"}]': finished 2024-03-11T20:57:02.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:02 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:02.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:02 smithi129 ceph-mon[30184]: osdmap e255: 8 total, 8 up, 8 in 2024-03-11T20:57:02.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:02 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-33"}]: dispatch 2024-03-11T20:57:02.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:02 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-33"}]: dispatch 2024-03-11T20:57:02.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:02 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi129-82555-40"}]: dispatch 2024-03-11T20:57:02.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:02 smithi129 ceph-mon[42066]: pgmap v335: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:02.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:02 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi129-82567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi129-82567-50"}]': finished 2024-03-11T20:57:02.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:02 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:02.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:02 smithi129 ceph-mon[42066]: osdmap e255: 8 total, 8 up, 8 in 2024-03-11T20:57:02.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:02 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-33"}]: dispatch 2024-03-11T20:57:02.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:02 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-33"}]: dispatch 2024-03-11T20:57:02.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:02 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi129-82555-40"}]: dispatch 2024-03-11T20:57:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:02 smithi157 ceph-mon[33359]: pgmap v335: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:02 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi129-82567-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi129-82567-50"}]': finished 2024-03-11T20:57:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:02 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:02 smithi157 ceph-mon[33359]: osdmap e255: 8 total, 8 up, 8 in 2024-03-11T20:57:02.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:02 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-33"}]: dispatch 2024-03-11T20:57:02.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:02 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-33"}]: dispatch 2024-03-11T20:57:02.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:02 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi129-82555-40"}]: dispatch 2024-03-11T20:57:03.721 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:57:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:57:03.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-33"}]': finished 2024-03-11T20:57:03.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi129-82555-40"}]': finished 2024-03-11T20:57:03.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[30184]: osdmap e256: 8 total, 8 up, 8 in 2024-03-11T20:57:03.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-33", "mode": "writeback"}]: dispatch 2024-03-11T20:57:03.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi129-82555-40"}]: dispatch 2024-03-11T20:57:03.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-33", "mode": "writeback"}]: dispatch 2024-03-11T20:57:03.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:03.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:03.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:57:03.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi129-82555-40"}]': finished 2024-03-11T20:57:03.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-33"}]': finished 2024-03-11T20:57:03.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi129-82555-40"}]': finished 2024-03-11T20:57:03.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[42066]: osdmap e256: 8 total, 8 up, 8 in 2024-03-11T20:57:03.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-33", "mode": "writeback"}]: dispatch 2024-03-11T20:57:03.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi129-82555-40"}]: dispatch 2024-03-11T20:57:03.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-33", "mode": "writeback"}]: dispatch 2024-03-11T20:57:03.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:03.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:03.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:57:03.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi129-82555-40"}]': finished 2024-03-11T20:57:03.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-33"}]': finished 2024-03-11T20:57:03.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi129-82555-40"}]': finished 2024-03-11T20:57:03.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:03 smithi157 ceph-mon[33359]: osdmap e256: 8 total, 8 up, 8 in 2024-03-11T20:57:03.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-33", "mode": "writeback"}]: dispatch 2024-03-11T20:57:03.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi129-82555-40"}]: dispatch 2024-03-11T20:57:03.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-33", "mode": "writeback"}]: dispatch 2024-03-11T20:57:03.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:03 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:03.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:03 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:03.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:03 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:57:03.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/667100390' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi129-82555-40"}]': finished 2024-03-11T20:57:04.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: pgmap v338: 300 pgs: 7 creating+peering, 1 unknown, 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:04.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:04.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-33", "mode": "writeback"}]': finished 2024-03-11T20:57:04.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: osdmap e257: 8 total, 8 up, 8 in 2024-03-11T20:57:04.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:04.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:04.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi129-82555-41"}]: dispatch 2024-03-11T20:57:04.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi129-82555-41"}]: dispatch 2024-03-11T20:57:04.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi129-82555-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:04.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:04.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:04.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:04.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi129-82567-50"}]': finished 2024-03-11T20:57:04.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi129-82555-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:04.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:57:04.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: osdmap e258: 8 total, 8 up, 8 in 2024-03-11T20:57:04.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi129-82555-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi129-82555-41"}]: dispatch 2024-03-11T20:57:04.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:04.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33"}]: dispatch 2024-03-11T20:57:04.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:04.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33"}]: dispatch 2024-03-11T20:57:04.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: pgmap v338: 300 pgs: 7 creating+peering, 1 unknown, 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:04.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:04.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-33", "mode": "writeback"}]': finished 2024-03-11T20:57:04.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: osdmap e257: 8 total, 8 up, 8 in 2024-03-11T20:57:04.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:04.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:04.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi129-82555-41"}]: dispatch 2024-03-11T20:57:04.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi129-82555-41"}]: dispatch 2024-03-11T20:57:04.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi129-82555-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:04.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:04.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:04.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:04.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi129-82567-50"}]': finished 2024-03-11T20:57:04.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi129-82555-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:04.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:57:04.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: osdmap e258: 8 total, 8 up, 8 in 2024-03-11T20:57:04.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi129-82555-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi129-82555-41"}]: dispatch 2024-03-11T20:57:04.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:04.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33"}]: dispatch 2024-03-11T20:57:04.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:04.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33"}]: dispatch 2024-03-11T20:57:04.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: pgmap v338: 300 pgs: 7 creating+peering, 1 unknown, 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:04.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:04.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-33", "mode": "writeback"}]': finished 2024-03-11T20:57:04.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: osdmap e257: 8 total, 8 up, 8 in 2024-03-11T20:57:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi129-82555-41"}]: dispatch 2024-03-11T20:57:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi129-82555-41"}]: dispatch 2024-03-11T20:57:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi129-82555-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi129-82567-50"}]': finished 2024-03-11T20:57:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi129-82555-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:57:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: osdmap e258: 8 total, 8 up, 8 in 2024-03-11T20:57:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi129-82555-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi129-82555-41"}]: dispatch 2024-03-11T20:57:04.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3097902731' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:04.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33"}]: dispatch 2024-03-11T20:57:04.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi129-82567-50"}]: dispatch 2024-03-11T20:57:04.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33"}]: dispatch 2024-03-11T20:57:06.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:57:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:57:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:57:06.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:06 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:57:06.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:06 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi129-82567-50"}]': finished 2024-03-11T20:57:06.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:06 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:57:06.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:06 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi129-82567-50"}]': finished 2024-03-11T20:57:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:06 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:57:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:06 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi129-82567-50"}]': finished 2024-03-11T20:57:07.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:07 smithi129 ceph-mon[30184]: pgmap v341: 292 pgs: 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:07.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33"}]': finished 2024-03-11T20:57:07.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:07 smithi129 ceph-mon[30184]: osdmap e259: 8 total, 8 up, 8 in 2024-03-11T20:57:07.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi129-82555-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi129-82555-41"}]': finished 2024-03-11T20:57:07.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:07 smithi129 ceph-mon[30184]: osdmap e260: 8 total, 8 up, 8 in 2024-03-11T20:57:07.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2188159475' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:07.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:07.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:07 smithi129 ceph-mon[42066]: pgmap v341: 292 pgs: 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:07.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33"}]': finished 2024-03-11T20:57:07.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:07 smithi129 ceph-mon[42066]: osdmap e259: 8 total, 8 up, 8 in 2024-03-11T20:57:07.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi129-82555-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi129-82555-41"}]': finished 2024-03-11T20:57:07.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:07 smithi129 ceph-mon[42066]: osdmap e260: 8 total, 8 up, 8 in 2024-03-11T20:57:07.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2188159475' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:07.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:07.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:07 smithi157 ceph-mon[33359]: pgmap v341: 292 pgs: 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:07.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-33"}]': finished 2024-03-11T20:57:07.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:07 smithi157 ceph-mon[33359]: osdmap e259: 8 total, 8 up, 8 in 2024-03-11T20:57:07.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi129-82555-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi129-82555-41"}]': finished 2024-03-11T20:57:07.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:07 smithi157 ceph-mon[33359]: osdmap e260: 8 total, 8 up, 8 in 2024-03-11T20:57:07.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2188159475' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:07.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:08.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:08 smithi129 ceph-mon[30184]: pgmap v344: 300 pgs: 35 creating+peering, 5 unknown, 260 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:57:08.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:08.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:08 smithi129 ceph-mon[30184]: osdmap e261: 8 total, 8 up, 8 in 2024-03-11T20:57:08.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:08.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:08.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:08 smithi129 ceph-mon[42066]: pgmap v344: 300 pgs: 35 creating+peering, 5 unknown, 260 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:57:08.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:08.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:08 smithi129 ceph-mon[42066]: osdmap e261: 8 total, 8 up, 8 in 2024-03-11T20:57:08.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:08.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:08.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:08 smithi157 ceph-mon[33359]: pgmap v344: 300 pgs: 35 creating+peering, 5 unknown, 260 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:57:08.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi129-82567-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:08.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:08 smithi157 ceph-mon[33359]: osdmap e261: 8 total, 8 up, 8 in 2024-03-11T20:57:08.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:08.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:09.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:09.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:09 smithi157 ceph-mon[33359]: osdmap e262: 8 total, 8 up, 8 in 2024-03-11T20:57:09.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi129-82555-41"}]: dispatch 2024-03-11T20:57:09.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:09.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:09.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi129-82567-52"}]: dispatch 2024-03-11T20:57:09.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi129-82567-52"}]: dispatch 2024-03-11T20:57:09.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi129-82567-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:09.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:09 smithi157 ceph-mon[33359]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:09.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi129-82555-41"}]': finished 2024-03-11T20:57:09.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:09.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi129-82567-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:09.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:09 smithi157 ceph-mon[33359]: osdmap e263: 8 total, 8 up, 8 in 2024-03-11T20:57:09.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi129-82555-41"}]: dispatch 2024-03-11T20:57:10.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:10.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[30184]: osdmap e262: 8 total, 8 up, 8 in 2024-03-11T20:57:10.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi129-82555-41"}]: dispatch 2024-03-11T20:57:10.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:10.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:10.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi129-82567-52"}]: dispatch 2024-03-11T20:57:10.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi129-82567-52"}]: dispatch 2024-03-11T20:57:10.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi129-82567-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:10.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[30184]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:10.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi129-82555-41"}]': finished 2024-03-11T20:57:10.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:10.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi129-82567-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:10.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[30184]: osdmap e263: 8 total, 8 up, 8 in 2024-03-11T20:57:10.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi129-82555-41"}]: dispatch 2024-03-11T20:57:10.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:10.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[42066]: osdmap e262: 8 total, 8 up, 8 in 2024-03-11T20:57:10.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi129-82555-41"}]: dispatch 2024-03-11T20:57:10.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:10.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:10.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi129-82567-52"}]: dispatch 2024-03-11T20:57:10.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi129-82567-52"}]: dispatch 2024-03-11T20:57:10.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi129-82567-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:10.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[42066]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:10.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi129-82555-41"}]': finished 2024-03-11T20:57:10.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:10.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi129-82567-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:10.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[42066]: osdmap e263: 8 total, 8 up, 8 in 2024-03-11T20:57:10.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi129-82555-41"}]: dispatch 2024-03-11T20:57:11.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:10 smithi129 ceph-mon[30184]: pgmap v347: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:57:11.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:11.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi129-82567-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi129-82567-52"}]: dispatch 2024-03-11T20:57:11.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:11.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi129-82555-41"}]': finished 2024-03-11T20:57:11.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:10 smithi129 ceph-mon[42066]: pgmap v347: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:57:11.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:11.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi129-82567-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi129-82567-52"}]: dispatch 2024-03-11T20:57:11.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:11.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi129-82555-41"}]': finished 2024-03-11T20:57:11.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:10 smithi157 ceph-mon[33359]: pgmap v347: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:57:11.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:11.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi129-82567-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi129-82567-52"}]: dispatch 2024-03-11T20:57:11.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:11.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3339023180' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi129-82555-41"}]': finished 2024-03-11T20:57:12.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]': finished 2024-03-11T20:57:12.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-35", "mode": "writeback"}]: dispatch 2024-03-11T20:57:12.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: pgmap v350: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:12.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: osdmap e264: 8 total, 8 up, 8 in 2024-03-11T20:57:12.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-35", "mode": "writeback"}]: dispatch 2024-03-11T20:57:12.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi129-82555-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:12.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi129-82555-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:12.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:57:12.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi129-82567-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi129-82567-52"}]': finished 2024-03-11T20:57:12.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-35", "mode": "writeback"}]': finished 2024-03-11T20:57:12.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi129-82555-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:12.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: osdmap e265: 8 total, 8 up, 8 in 2024-03-11T20:57:12.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi129-82555-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi129-82555-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]': finished 2024-03-11T20:57:12.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-35", "mode": "writeback"}]: dispatch 2024-03-11T20:57:12.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: pgmap v350: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:12.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: osdmap e264: 8 total, 8 up, 8 in 2024-03-11T20:57:12.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-35", "mode": "writeback"}]: dispatch 2024-03-11T20:57:12.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi129-82555-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:12.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi129-82555-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:12.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:57:12.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi129-82567-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi129-82567-52"}]': finished 2024-03-11T20:57:12.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-35", "mode": "writeback"}]': finished 2024-03-11T20:57:12.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi129-82555-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:12.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: osdmap e265: 8 total, 8 up, 8 in 2024-03-11T20:57:12.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi129-82555-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi129-82555-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]': finished 2024-03-11T20:57:12.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-35", "mode": "writeback"}]: dispatch 2024-03-11T20:57:12.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: pgmap v350: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 568 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:12.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: osdmap e264: 8 total, 8 up, 8 in 2024-03-11T20:57:12.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-35", "mode": "writeback"}]: dispatch 2024-03-11T20:57:12.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi129-82555-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:12.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi129-82555-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:12.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:57:12.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi129-82567-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi129-82567-52"}]': finished 2024-03-11T20:57:12.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-35", "mode": "writeback"}]': finished 2024-03-11T20:57:12.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi129-82555-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:12.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: osdmap e265: 8 total, 8 up, 8 in 2024-03-11T20:57:12.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi129-82555-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:12.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi129-82555-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:13.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:57:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:57:14.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:13 smithi129 ceph-mon[30184]: osdmap e266: 8 total, 8 up, 8 in 2024-03-11T20:57:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:13 smithi129 ceph-mon[42066]: osdmap e266: 8 total, 8 up, 8 in 2024-03-11T20:57:14.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:13 smithi157 ceph-mon[33359]: osdmap e266: 8 total, 8 up, 8 in 2024-03-11T20:57:15.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:14 smithi129 ceph-mon[30184]: pgmap v353: 300 pgs: 5 creating+peering, 3 unknown, 292 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:57:15.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:14 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:15.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi129-82555-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi129-82555-42"}]': finished 2024-03-11T20:57:15.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:14 smithi129 ceph-mon[30184]: osdmap e267: 8 total, 8 up, 8 in 2024-03-11T20:57:15.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi129-82567-52"}]: dispatch 2024-03-11T20:57:15.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:15.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:15.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:14 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:14 smithi129 ceph-mon[42066]: pgmap v353: 300 pgs: 5 creating+peering, 3 unknown, 292 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:57:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:14 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi129-82555-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi129-82555-42"}]': finished 2024-03-11T20:57:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:14 smithi129 ceph-mon[42066]: osdmap e267: 8 total, 8 up, 8 in 2024-03-11T20:57:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi129-82567-52"}]: dispatch 2024-03-11T20:57:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:15.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:14 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:14 smithi157 ceph-mon[33359]: pgmap v353: 300 pgs: 5 creating+peering, 3 unknown, 292 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:57:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:14 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi129-82555-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi129-82555-42"}]': finished 2024-03-11T20:57:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:14 smithi157 ceph-mon[33359]: osdmap e267: 8 total, 8 up, 8 in 2024-03-11T20:57:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi129-82567-52"}]: dispatch 2024-03-11T20:57:15.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:15.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:15.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:14 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi129-82567-52"}]': finished 2024-03-11T20:57:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:57:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:15 smithi129 ceph-mon[30184]: osdmap e268: 8 total, 8 up, 8 in 2024-03-11T20:57:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi129-82567-52"}]: dispatch 2024-03-11T20:57:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:15 smithi129 ceph-mon[30184]: pgmap v356: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:57:16.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi129-82567-52"}]': finished 2024-03-11T20:57:16.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi129-82567-52"}]': finished 2024-03-11T20:57:16.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:57:16.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:15 smithi129 ceph-mon[42066]: osdmap e268: 8 total, 8 up, 8 in 2024-03-11T20:57:16.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi129-82567-52"}]: dispatch 2024-03-11T20:57:16.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:16.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:16.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:15 smithi129 ceph-mon[42066]: pgmap v356: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:57:16.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi129-82567-52"}]': finished 2024-03-11T20:57:16.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi129-82567-52"}]': finished 2024-03-11T20:57:16.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:57:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:15 smithi157 ceph-mon[33359]: osdmap e268: 8 total, 8 up, 8 in 2024-03-11T20:57:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi129-82567-52"}]: dispatch 2024-03-11T20:57:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:15 smithi157 ceph-mon[33359]: pgmap v356: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:57:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1641070464' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi129-82567-52"}]': finished 2024-03-11T20:57:16.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:57:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:57:17.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]': finished 2024-03-11T20:57:17.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[42066]: osdmap e269: 8 total, 8 up, 8 in 2024-03-11T20:57:17.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:17.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:17.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi129-82567-53"}]: dispatch 2024-03-11T20:57:17.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi129-82567-53"}]: dispatch 2024-03-11T20:57:17.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi129-82567-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:17.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi129-82555-42"}]': finished 2024-03-11T20:57:17.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi129-82567-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:17.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[42066]: osdmap e270: 8 total, 8 up, 8 in 2024-03-11T20:57:17.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi129-82567-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi129-82567-53"}]: dispatch 2024-03-11T20:57:17.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:17.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:17.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]': finished 2024-03-11T20:57:17.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[30184]: osdmap e269: 8 total, 8 up, 8 in 2024-03-11T20:57:17.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:17.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:17.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi129-82567-53"}]: dispatch 2024-03-11T20:57:17.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi129-82567-53"}]: dispatch 2024-03-11T20:57:17.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi129-82567-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:17.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi129-82555-42"}]': finished 2024-03-11T20:57:17.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi129-82567-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:17.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[30184]: osdmap e270: 8 total, 8 up, 8 in 2024-03-11T20:57:17.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi129-82567-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi129-82567-53"}]: dispatch 2024-03-11T20:57:17.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:17.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:16 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:17.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:16 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-35"}]': finished 2024-03-11T20:57:17.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:16 smithi157 ceph-mon[33359]: osdmap e269: 8 total, 8 up, 8 in 2024-03-11T20:57:17.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:17.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:16 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:17.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi129-82567-53"}]: dispatch 2024-03-11T20:57:17.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi129-82567-53"}]: dispatch 2024-03-11T20:57:17.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi129-82567-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:17.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:16 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi129-82555-42"}]': finished 2024-03-11T20:57:17.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi129-82567-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:17.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:16 smithi157 ceph-mon[33359]: osdmap e270: 8 total, 8 up, 8 in 2024-03-11T20:57:17.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi129-82567-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi129-82567-53"}]: dispatch 2024-03-11T20:57:17.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4223145144' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:17.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:16 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi129-82555-42"}]: dispatch 2024-03-11T20:57:17.760 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: sta api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6776 ms) 2024-03-11T20:57:17.760 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-03-11T20:57:17.760 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6961 ms) 2024-03-11T20:57:17.760 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-03-11T20:57:17.760 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7018 ms) 2024-03-11T20:57:17.760 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-03-11T20:57:17.760 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7034 ms) 2024-03-11T20:57:17.760 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-03-11T20:57:17.760 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7041 ms) 2024-03-11T20:57:17.761 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-03-11T20:57:17.761 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7040 ms) 2024-03-11T20:57:17.761 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-03-11T20:57:17.761 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7052 ms) 2024-03-11T20:57:17.761 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-03-11T20:57:17.761 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6836 ms) 2024-03-11T20:57:17.761 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-03-11T20:57:17.761 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7042 ms) 2024-03-11T20:57:17.761 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-03-11T20:57:17.761 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7025 ms) 2024-03-11T20:57:17.761 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-03-11T20:57:17.761 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7034 ms) 2024-03-11T20:57:17.761 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-03-11T20:57:17.761 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6920 ms) 2024-03-11T20:57:17.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-03-11T20:57:17.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6986 ms) 2024-03-11T20:57:17.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-03-11T20:57:17.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7024 ms) 2024-03-11T20:57:17.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-03-11T20:57:17.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7258 ms) 2024-03-11T20:57:17.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-03-11T20:57:17.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6785 ms) 2024-03-11T20:57:17.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (111832 ms total) 2024-03-11T20:57:17.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: 2024-03-11T20:57:17.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [----------] Global test environment tear-down 2024-03-11T20:57:17.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (204096 ms total) 2024-03-11T20:57:17.762 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio: [ PASSED ] 42 tests. 2024-03-11T20:57:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:18 smithi129 ceph-mon[30184]: pgmap v359: 292 pgs: 2 active+clean+snaptrim_wait, 290 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-11T20:57:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi129-82555-42"}]': finished 2024-03-11T20:57:18.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:18.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:18.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:18 smithi129 ceph-mon[42066]: pgmap v359: 292 pgs: 2 active+clean+snaptrim_wait, 290 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-11T20:57:18.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi129-82555-42"}]': finished 2024-03-11T20:57:18.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:18.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:18.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:18 smithi157 ceph-mon[33359]: pgmap v359: 292 pgs: 2 active+clean+snaptrim_wait, 290 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-11T20:57:18.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi129-82555-42"}]': finished 2024-03-11T20:57:19.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:57:19.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:19.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[30184]: osdmap e271: 8 total, 8 up, 8 in 2024-03-11T20:57:19.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:19.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:19.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:19.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:57:19.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi129-82567-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi129-82567-53"}]': finished 2024-03-11T20:57:19.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35"}]': finished 2024-03-11T20:57:19.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[30184]: osdmap e272: 8 total, 8 up, 8 in 2024-03-11T20:57:19.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:57:19.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:19.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[42066]: osdmap e271: 8 total, 8 up, 8 in 2024-03-11T20:57:19.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:19.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:19.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:19.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:57:19.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi129-82567-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi129-82567-53"}]': finished 2024-03-11T20:57:19.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35"}]': finished 2024-03-11T20:57:19.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:19 smithi129 ceph-mon[42066]: osdmap e272: 8 total, 8 up, 8 in 2024-03-11T20:57:19.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:57:19.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:19.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:19 smithi157 ceph-mon[33359]: osdmap e271: 8 total, 8 up, 8 in 2024-03-11T20:57:19.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35"}]: dispatch 2024-03-11T20:57:19.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:19.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:19.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:19 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:57:19.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi129-82567-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi129-82567-53"}]': finished 2024-03-11T20:57:19.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-35"}]': finished 2024-03-11T20:57:19.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:19 smithi157 ceph-mon[33359]: osdmap e272: 8 total, 8 up, 8 in 2024-03-11T20:57:20.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:20 smithi129 ceph-mon[30184]: pgmap v362: 300 pgs: 8 unknown, 2 active+clean+snaptrim_wait, 290 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-11T20:57:20.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:20 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:20.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:20 smithi129 ceph-mon[30184]: osdmap e273: 8 total, 8 up, 8 in 2024-03-11T20:57:20.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:20 smithi129 ceph-mon[42066]: pgmap v362: 300 pgs: 8 unknown, 2 active+clean+snaptrim_wait, 290 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-11T20:57:20.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:20 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:20.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:20 smithi129 ceph-mon[42066]: osdmap e273: 8 total, 8 up, 8 in 2024-03-11T20:57:20.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:20 smithi157 ceph-mon[33359]: pgmap v362: 300 pgs: 8 unknown, 2 active+clean+snaptrim_wait, 290 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-11T20:57:20.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:20 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:20.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:20 smithi157 ceph-mon[33359]: osdmap e273: 8 total, 8 up, 8 in 2024-03-11T20:57:22.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:21 smithi129 ceph-mon[30184]: osdmap e274: 8 total, 8 up, 8 in 2024-03-11T20:57:22.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:22.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:22.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi129-82567-53"}]: dispatch 2024-03-11T20:57:22.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:21 smithi129 ceph-mon[30184]: pgmap v365: 292 pgs: 32 unknown, 1 active+clean+snaptrim_wait, 259 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:22.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:21 smithi129 ceph-mon[42066]: osdmap e274: 8 total, 8 up, 8 in 2024-03-11T20:57:22.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:22.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:22.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi129-82567-53"}]: dispatch 2024-03-11T20:57:22.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:21 smithi129 ceph-mon[42066]: pgmap v365: 292 pgs: 32 unknown, 1 active+clean+snaptrim_wait, 259 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:22.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:21 smithi157 ceph-mon[33359]: osdmap e274: 8 total, 8 up, 8 in 2024-03-11T20:57:22.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:22.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:22.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi129-82567-53"}]: dispatch 2024-03-11T20:57:22.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:21 smithi157 ceph-mon[33359]: pgmap v365: 292 pgs: 32 unknown, 1 active+clean+snaptrim_wait, 259 active+clean; 4.6 MiB data, 573 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:23.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:23.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi129-82567-53"}]': finished 2024-03-11T20:57:23.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[30184]: osdmap e275: 8 total, 8 up, 8 in 2024-03-11T20:57:23.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi129-82567-53"}]: dispatch 2024-03-11T20:57:23.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:23.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:23.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi129-82567-53"}]': finished 2024-03-11T20:57:23.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:23.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[30184]: osdmap e276: 8 total, 8 up, 8 in 2024-03-11T20:57:23.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-37"}]: dispatch 2024-03-11T20:57:23.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-37"}]: dispatch 2024-03-11T20:57:23.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:23.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi129-82567-53"}]': finished 2024-03-11T20:57:23.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[42066]: osdmap e275: 8 total, 8 up, 8 in 2024-03-11T20:57:23.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi129-82567-53"}]: dispatch 2024-03-11T20:57:23.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:23.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:23.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi129-82567-53"}]': finished 2024-03-11T20:57:23.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:23.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[42066]: osdmap e276: 8 total, 8 up, 8 in 2024-03-11T20:57:23.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-37"}]: dispatch 2024-03-11T20:57:23.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-37"}]: dispatch 2024-03-11T20:57:23.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:23.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi129-82567-53"}]': finished 2024-03-11T20:57:23.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:22 smithi157 ceph-mon[33359]: osdmap e275: 8 total, 8 up, 8 in 2024-03-11T20:57:23.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi129-82567-53"}]: dispatch 2024-03-11T20:57:23.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:23.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4234381166' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi129-82567-53"}]': finished 2024-03-11T20:57:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:22 smithi157 ceph-mon[33359]: osdmap e276: 8 total, 8 up, 8 in 2024-03-11T20:57:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-37"}]: dispatch 2024-03-11T20:57:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-37"}]: dispatch 2024-03-11T20:57:23.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:57:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:57:24.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:24.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:24.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:24.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:24.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi129-82567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:24.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi129-82567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:24.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[30184]: pgmap v368: 292 pgs: 20 creating+activating, 1 active, 271 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T20:57:24.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:24.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-37"}]': finished 2024-03-11T20:57:24.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi129-82567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:24.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[30184]: osdmap e277: 8 total, 8 up, 8 in 2024-03-11T20:57:24.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:24.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:24.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:24.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:24.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi129-82567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:24.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi129-82567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:24.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[42066]: pgmap v368: 292 pgs: 20 creating+activating, 1 active, 271 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T20:57:24.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:24.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-37"}]': finished 2024-03-11T20:57:24.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi129-82567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:24.233 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:23 smithi129 ceph-mon[42066]: osdmap e277: 8 total, 8 up, 8 in 2024-03-11T20:57:24.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:24.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:23 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:23 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi129-82567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:23 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi129-82567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:23 smithi157 ceph-mon[33359]: pgmap v368: 292 pgs: 20 creating+activating, 1 active, 271 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T20:57:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:23 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:23 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-37"}]': finished 2024-03-11T20:57:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:23 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi129-82567-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:23 smithi157 ceph-mon[33359]: osdmap e277: 8 total, 8 up, 8 in 2024-03-11T20:57:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-37", "mode": "writeback"}]: dispatch 2024-03-11T20:57:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi129-82567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-37", "mode": "writeback"}]: dispatch 2024-03-11T20:57:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi129-82567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:24 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:57:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-37", "mode": "writeback"}]': finished 2024-03-11T20:57:25.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-37", "mode": "writeback"}]: dispatch 2024-03-11T20:57:25.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi129-82567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:25.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-37", "mode": "writeback"}]: dispatch 2024-03-11T20:57:25.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi129-82567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:25.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:24 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:57:25.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-37", "mode": "writeback"}]': finished 2024-03-11T20:57:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-37", "mode": "writeback"}]: dispatch 2024-03-11T20:57:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi129-82567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-37", "mode": "writeback"}]: dispatch 2024-03-11T20:57:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi129-82567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:24 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:57:25.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-37", "mode": "writeback"}]': finished 2024-03-11T20:57:26.229 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:57:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:57:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:57:26.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:26.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[30184]: pgmap v371: 292 pgs: 20 creating+activating, 1 active, 271 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:26.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[30184]: osdmap e278: 8 total, 8 up, 8 in 2024-03-11T20:57:26.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:26.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi129-82567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi129-82567-54"}]': finished 2024-03-11T20:57:26.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:57:26.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[30184]: osdmap e279: 8 total, 8 up, 8 in 2024-03-11T20:57:26.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37"}]: dispatch 2024-03-11T20:57:26.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37"}]: dispatch 2024-03-11T20:57:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[42066]: pgmap v371: 292 pgs: 20 creating+activating, 1 active, 271 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[42066]: osdmap e278: 8 total, 8 up, 8 in 2024-03-11T20:57:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi129-82567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi129-82567-54"}]': finished 2024-03-11T20:57:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:57:26.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[42066]: osdmap e279: 8 total, 8 up, 8 in 2024-03-11T20:57:26.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37"}]: dispatch 2024-03-11T20:57:26.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37"}]: dispatch 2024-03-11T20:57:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:25 smithi157 ceph-mon[33359]: pgmap v371: 292 pgs: 20 creating+activating, 1 active, 271 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:25 smithi157 ceph-mon[33359]: osdmap e278: 8 total, 8 up, 8 in 2024-03-11T20:57:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi129-82567-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi129-82567-54"}]': finished 2024-03-11T20:57:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:57:26.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:25 smithi157 ceph-mon[33359]: osdmap e279: 8 total, 8 up, 8 in 2024-03-11T20:57:26.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37"}]: dispatch 2024-03-11T20:57:26.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37"}]: dispatch 2024-03-11T20:57:27.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:26 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:57:27.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:26 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37"}]': finished 2024-03-11T20:57:27.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:26 smithi129 ceph-mon[30184]: osdmap e280: 8 total, 8 up, 8 in 2024-03-11T20:57:27.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:26 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:57:27.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:26 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37"}]': finished 2024-03-11T20:57:27.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:26 smithi129 ceph-mon[42066]: osdmap e280: 8 total, 8 up, 8 in 2024-03-11T20:57:27.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:26 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:57:27.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:26 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-37"}]': finished 2024-03-11T20:57:27.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:26 smithi157 ceph-mon[33359]: osdmap e280: 8 total, 8 up, 8 in 2024-03-11T20:57:28.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:27 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:28.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:27 smithi129 ceph-mon[30184]: osdmap e281: 8 total, 8 up, 8 in 2024-03-11T20:57:28.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:28.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:27 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:28.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:27 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:28.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:27 smithi129 ceph-mon[42066]: osdmap e281: 8 total, 8 up, 8 in 2024-03-11T20:57:28.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:28.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:27 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:28.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:27 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:28.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:27 smithi157 ceph-mon[33359]: osdmap e281: 8 total, 8 up, 8 in 2024-03-11T20:57:28.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:28.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:27 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:29.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:28 smithi129 ceph-mon[42066]: pgmap v374: 300 pgs: 1 unknown, 299 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-03-11T20:57:29.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi129-82567-54"}]': finished 2024-03-11T20:57:29.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:28 smithi129 ceph-mon[42066]: osdmap e282: 8 total, 8 up, 8 in 2024-03-11T20:57:29.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:29.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:29.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:29.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:29.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:28 smithi129 ceph-mon[30184]: pgmap v374: 300 pgs: 1 unknown, 299 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-03-11T20:57:29.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi129-82567-54"}]': finished 2024-03-11T20:57:29.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:28 smithi129 ceph-mon[30184]: osdmap e282: 8 total, 8 up, 8 in 2024-03-11T20:57:29.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:29.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:29.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:29.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:29.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:28 smithi157 ceph-mon[33359]: pgmap v374: 300 pgs: 1 unknown, 299 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-03-11T20:57:29.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi129-82567-54"}]': finished 2024-03-11T20:57:29.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:28 smithi157 ceph-mon[33359]: osdmap e282: 8 total, 8 up, 8 in 2024-03-11T20:57:29.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2325381697' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:29.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi129-82567-54"}]: dispatch 2024-03-11T20:57:29.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:29.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:30.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[30184]: pgmap v377: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T20:57:30.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi129-82567-54"}]': finished 2024-03-11T20:57:30.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:30.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[30184]: osdmap e283: 8 total, 8 up, 8 in 2024-03-11T20:57:30.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:30.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:30.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:30.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:30.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:30.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:30.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi129-82567-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:30.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi129-82567-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:30.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[42066]: pgmap v377: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T20:57:30.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi129-82567-54"}]': finished 2024-03-11T20:57:30.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:30.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[42066]: osdmap e283: 8 total, 8 up, 8 in 2024-03-11T20:57:30.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:30.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:30.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:30.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:30.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:30.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:30.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi129-82567-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:30.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:30 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi129-82567-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:30 smithi157 ceph-mon[33359]: pgmap v377: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T20:57:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:30 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi129-82567-54"}]': finished 2024-03-11T20:57:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:30 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:30.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:30 smithi157 ceph-mon[33359]: osdmap e283: 8 total, 8 up, 8 in 2024-03-11T20:57:30.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:30.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:30 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:30.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:30.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:30 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:30.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:30.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:30 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:30.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi129-82567-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:30.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:30 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi129-82567-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:31.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:31.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi129-82567-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:31.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:31 smithi129 ceph-mon[30184]: osdmap e284: 8 total, 8 up, 8 in 2024-03-11T20:57:31.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-39"}]: dispatch 2024-03-11T20:57:31.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi129-82567-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:31.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-39"}]: dispatch 2024-03-11T20:57:31.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi129-82567-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:31.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:31.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi129-82567-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:31.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:31 smithi129 ceph-mon[42066]: osdmap e284: 8 total, 8 up, 8 in 2024-03-11T20:57:31.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-39"}]: dispatch 2024-03-11T20:57:31.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi129-82567-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:31.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-39"}]: dispatch 2024-03-11T20:57:31.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi129-82567-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi129-82567-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:31 smithi157 ceph-mon[33359]: osdmap e284: 8 total, 8 up, 8 in 2024-03-11T20:57:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-39"}]: dispatch 2024-03-11T20:57:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi129-82567-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-39"}]: dispatch 2024-03-11T20:57:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi129-82567-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:32 smithi129 ceph-mon[30184]: pgmap v380: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:32 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-39"}]': finished 2024-03-11T20:57:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:32 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-39", "mode": "writeback"}]: dispatch 2024-03-11T20:57:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:32 smithi129 ceph-mon[30184]: osdmap e285: 8 total, 8 up, 8 in 2024-03-11T20:57:33.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:32 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-39", "mode": "writeback"}]: dispatch 2024-03-11T20:57:33.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:32 smithi129 ceph-mon[42066]: pgmap v380: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:33.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:32 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-39"}]': finished 2024-03-11T20:57:33.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:32 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-39", "mode": "writeback"}]: dispatch 2024-03-11T20:57:33.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:32 smithi129 ceph-mon[42066]: osdmap e285: 8 total, 8 up, 8 in 2024-03-11T20:57:33.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:32 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-39", "mode": "writeback"}]: dispatch 2024-03-11T20:57:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:32 smithi157 ceph-mon[33359]: pgmap v380: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:32 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-39"}]': finished 2024-03-11T20:57:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:32 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-39", "mode": "writeback"}]: dispatch 2024-03-11T20:57:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:32 smithi157 ceph-mon[33359]: osdmap e285: 8 total, 8 up, 8 in 2024-03-11T20:57:33.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:32 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-39", "mode": "writeback"}]: dispatch 2024-03-11T20:57:33.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:57:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:57:34.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:57:34.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi129-82567-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi129-82567-55"}]': finished 2024-03-11T20:57:34.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-39", "mode": "writeback"}]': finished 2024-03-11T20:57:34.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[30184]: osdmap e286: 8 total, 8 up, 8 in 2024-03-11T20:57:34.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:34.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:34.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:34.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:57:34.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[30184]: osdmap e287: 8 total, 8 up, 8 in 2024-03-11T20:57:34.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39"}]: dispatch 2024-03-11T20:57:34.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39"}]: dispatch 2024-03-11T20:57:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:57:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi129-82567-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi129-82567-55"}]': finished 2024-03-11T20:57:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-39", "mode": "writeback"}]': finished 2024-03-11T20:57:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[42066]: osdmap e286: 8 total, 8 up, 8 in 2024-03-11T20:57:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:34.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:34.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:57:34.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[42066]: osdmap e287: 8 total, 8 up, 8 in 2024-03-11T20:57:34.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39"}]: dispatch 2024-03-11T20:57:34.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39"}]: dispatch 2024-03-11T20:57:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:33 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:57:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi129-82567-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi129-82567-55"}]': finished 2024-03-11T20:57:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-39", "mode": "writeback"}]': finished 2024-03-11T20:57:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:33 smithi157 ceph-mon[33359]: osdmap e286: 8 total, 8 up, 8 in 2024-03-11T20:57:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:34.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:33 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:57:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:33 smithi157 ceph-mon[33359]: osdmap e287: 8 total, 8 up, 8 in 2024-03-11T20:57:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39"}]: dispatch 2024-03-11T20:57:34.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39"}]: dispatch 2024-03-11T20:57:35.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:34 smithi129 ceph-mon[30184]: pgmap v383: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:35.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:34 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:35.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:34 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:35.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:34 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:57:35.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:34 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39"}]': finished 2024-03-11T20:57:35.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:34 smithi129 ceph-mon[30184]: osdmap e288: 8 total, 8 up, 8 in 2024-03-11T20:57:35.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:34 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:35.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:34 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:35.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:34 smithi129 ceph-mon[42066]: pgmap v383: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:35.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:34 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:35.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:34 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:35.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:34 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:57:35.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:34 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39"}]': finished 2024-03-11T20:57:35.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:34 smithi129 ceph-mon[42066]: osdmap e288: 8 total, 8 up, 8 in 2024-03-11T20:57:35.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:34 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:35.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:34 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:35.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:34 smithi157 ceph-mon[33359]: pgmap v383: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:35.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:34 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:35.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:34 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:35.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:34 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:57:35.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:34 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-39"}]': finished 2024-03-11T20:57:35.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:34 smithi157 ceph-mon[33359]: osdmap e288: 8 total, 8 up, 8 in 2024-03-11T20:57:35.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:34 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:35.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:34 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:35.737 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-03-11T20:57:35.737 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-03-11T20:57:35.738 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-03-11T20:57:35.738 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: seed 82818 2024-03-11T20:57:35.738 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-03-11T20:57:35.738 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (877 ms) 2024-03-11T20:57:35.738 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-03-11T20:57:35.738 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (16199 ms) 2024-03-11T20:57:35.738 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-03-11T20:57:35.738 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (29 ms) 2024-03-11T20:57:35.738 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (17106 ms total) 2024-03-11T20:57:35.738 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 2024-03-11T20:57:35.738 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-03-11T20:57:35.738 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-03-11T20:57:35.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (8233 ms) 2024-03-11T20:57:35.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-03-11T20:57:35.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8267 ms) 2024-03-11T20:57:35.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-03-11T20:57:35.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9765 ms) 2024-03-11T20:57:35.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-03-11T20:57:35.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: my_snaps [3] 2024-03-11T20:57:35.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: my_snaps [4,3] 2024-03-11T20:57:35.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: my_snaps [5,4,3] 2024-03-11T20:57:35.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-03-11T20:57:35.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: promoting some heads 2024-03-11T20:57:35.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: promoting from clones for snap 6 2024-03-11T20:57:35.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: promoting from clones for snap 5 2024-03-11T20:57:35.739 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: promoting from clones for snap 4 2024-03-11T20:57:35.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: promoting from clones for snap 3 2024-03-11T20:57:35.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: waiting for scrubs... 2024-03-11T20:57:35.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: done waiting 2024-03-11T20:57:35.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46252 ms) 2024-03-11T20:57:35.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-03-11T20:57:35.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9999 ms) 2024-03-11T20:57:35.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-03-11T20:57:35.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7804 ms) 2024-03-11T20:57:35.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-03-11T20:57:35.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8038 ms) 2024-03-11T20:57:35.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-03-11T20:57:35.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8048 ms) 2024-03-11T20:57:35.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-03-11T20:57:35.740 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10056 ms) 2024-03-11T20:57:35.741 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-03-11T20:57:35.741 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8859 ms) 2024-03-11T20:57:35.741 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-03-11T20:57:35.741 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10290 ms) 2024-03-11T20:57:35.741 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-03-11T20:57:35.741 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12697 ms) 2024-03-11T20:57:35.741 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-03-11T20:57:35.741 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (8235 ms) 2024-03-11T20:57:35.741 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-03-11T20:57:35.741 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (7780 ms) 2024-03-11T20:57:35.741 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-03-11T20:57:35.741 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13040 ms) 2024-03-11T20:57:35.741 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-03-11T20:57:35.741 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7949 ms) 2024-03-11T20:57:35.742 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-03-11T20:57:35.742 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8030 ms) 2024-03-11T20:57:36.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:57:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:57:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:57:37.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:36 smithi129 ceph-mon[30184]: pgmap v386: 292 pgs: 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:37.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi129-82567-55"}]': finished 2024-03-11T20:57:37.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:36 smithi129 ceph-mon[30184]: osdmap e289: 8 total, 8 up, 8 in 2024-03-11T20:57:37.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:37.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:37.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:36 smithi129 ceph-mon[42066]: pgmap v386: 292 pgs: 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:37.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi129-82567-55"}]': finished 2024-03-11T20:57:37.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:36 smithi129 ceph-mon[42066]: osdmap e289: 8 total, 8 up, 8 in 2024-03-11T20:57:37.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:37.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:37.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:36 smithi157 ceph-mon[33359]: pgmap v386: 292 pgs: 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:37.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi129-82567-55"}]': finished 2024-03-11T20:57:37.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:36 smithi157 ceph-mon[33359]: osdmap e289: 8 total, 8 up, 8 in 2024-03-11T20:57:37.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2069312705' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:37.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi129-82567-55"}]: dispatch 2024-03-11T20:57:38.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi129-82567-55"}]': finished 2024-03-11T20:57:38.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[30184]: osdmap e290: 8 total, 8 up, 8 in 2024-03-11T20:57:38.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:38.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:38.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:38.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:38.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:38.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:38.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi129-82567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:38.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi129-82567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:38.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[30184]: pgmap v389: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-11T20:57:38.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:38.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi129-82567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:38.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[30184]: osdmap e291: 8 total, 8 up, 8 in 2024-03-11T20:57:38.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi129-82567-55"}]': finished 2024-03-11T20:57:38.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[42066]: osdmap e290: 8 total, 8 up, 8 in 2024-03-11T20:57:38.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:38.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:38.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:38.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:38.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:38.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:38.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi129-82567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:38.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi129-82567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:38.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[42066]: pgmap v389: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-11T20:57:38.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:38.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi129-82567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:38.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:37 smithi129 ceph-mon[42066]: osdmap e291: 8 total, 8 up, 8 in 2024-03-11T20:57:38.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi129-82567-55"}]': finished 2024-03-11T20:57:38.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:37 smithi157 ceph-mon[33359]: osdmap e290: 8 total, 8 up, 8 in 2024-03-11T20:57:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi129-82567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi129-82567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:37 smithi157 ceph-mon[33359]: pgmap v389: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-03-11T20:57:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi129-82567-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:38.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:37 smithi157 ceph-mon[33359]: osdmap e291: 8 total, 8 up, 8 in 2024-03-11T20:57:39.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:39.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi129-82567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:39.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:39.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi129-82567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:39.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:39.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:39.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi129-82567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:39.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:39.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi129-82567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:39.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:39.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:39.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi129-82567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:39.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:39.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi129-82567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:39.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:40.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetrted 0x558254a395b0 2024-03-11T20:57:40.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a39cd0 2024-03-11T20:57:40.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a39820 2024-03-11T20:57:40.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a39950 2024-03-11T20:57:40.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a39a60 2024-03-11T20:57:40.122 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a39bb0 2024-03-11T20:57:40.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3a2d0 2024-03-11T20:57:40.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a39e20 2024-03-11T20:57:40.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a39f70 2024-03-11T20:57:40.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3a080 2024-03-11T20:57:40.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3a420 2024-03-11T20:57:40.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3a8f0 2024-03-11T20:57:40.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3aa40 2024-03-11T20:57:40.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3a530 2024-03-11T20:57:40.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3a680 2024-03-11T20:57:40.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3a7d0 2024-03-11T20:57:40.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3ab70 2024-03-11T20:57:40.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3b060 2024-03-11T20:57:40.123 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3b170 2024-03-11T20:57:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3b730 2024-03-11T20:57:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3acc0 2024-03-11T20:57:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3adf0 2024-03-11T20:57:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3af40 2024-03-11T20:57:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3b880 2024-03-11T20:57:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3b280 2024-03-11T20:57:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3b3b0 2024-03-11T20:57:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3b4e0 2024-03-11T20:57:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3b610 2024-03-11T20:57:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3be80 2024-03-11T20:57:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3b9d0 2024-03-11T20:57:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3bb00 2024-03-11T20:57:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3bc10 2024-03-11T20:57:40.124 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3bd60 2024-03-11T20:57:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3c460 2024-03-11T20:57:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3bfb0 2024-03-11T20:57:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3c100 2024-03-11T20:57:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3c230 2024-03-11T20:57:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3c5b0 2024-03-11T20:57:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3caa0 2024-03-11T20:57:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3cbb0 2024-03-11T20:57:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3d190 2024-03-11T20:57:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3c700 2024-03-11T20:57:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3c830 2024-03-11T20:57:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3c960 2024-03-11T20:57:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3d2a0 2024-03-11T20:57:40.125 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3cce0 2024-03-11T20:57:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3ce30 2024-03-11T20:57:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3cf60 2024-03-11T20:57:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3d3b0 2024-03-11T20:57:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3d880 2024-03-11T20:57:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3d9f0 2024-03-11T20:57:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3d070 2024-03-11T20:57:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3db20 2024-03-11T20:57:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3dc50 2024-03-11T20:57:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3d4c0 2024-03-11T20:57:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3d5d0 2024-03-11T20:57:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3d720 2024-03-11T20:57:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3dd80 2024-03-11T20:57:40.126 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3df00 2024-03-11T20:57:40.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3e500 2024-03-11T20:57:40.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3e050 2024-03-11T20:57:40.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3e1a0 2024-03-11T20:57:40.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3e310 2024-03-11T20:57:40.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3e610 2024-03-11T20:57:40.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3eb00 2024-03-11T20:57:40.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3ec30 2024-03-11T20:57:40.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a38730 2024-03-11T20:57:40.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a38860 2024-03-11T20:57:40.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a38970 2024-03-11T20:57:40.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3ed60 2024-03-11T20:57:40.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3f210 2024-03-11T20:57:40.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3f360 2024-03-11T20:57:40.127 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3f940 2024-03-11T20:57:40.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3ee70 2024-03-11T20:57:40.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3efa0 2024-03-11T20:57:40.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3f0d0 2024-03-11T20:57:40.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3fa70 2024-03-11T20:57:40.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3f490 2024-03-11T20:57:40.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3f5c0 2024-03-11T20:57:40.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3f6d0 2024-03-11T20:57:40.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3f800 2024-03-11T20:57:40.128 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a40050 2024-03-11T20:57:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:57:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[30184]: pgmap v392: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-11T20:57:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[30184]: osdmap e292: 8 total, 8 up, 8 in 2024-03-11T20:57:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:57:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi129-82567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi129-82567-56"}]': finished 2024-03-11T20:57:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_count","val": "2"}]': finished 2024-03-11T20:57:40.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[30184]: osdmap e293: 8 total, 8 up, 8 in 2024-03-11T20:57:40.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:57:40.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:57:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:57:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[42066]: pgmap v392: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-11T20:57:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[42066]: osdmap e292: 8 total, 8 up, 8 in 2024-03-11T20:57:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:57:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi129-82567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi129-82567-56"}]': finished 2024-03-11T20:57:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_count","val": "2"}]': finished 2024-03-11T20:57:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[42066]: osdmap e293: 8 total, 8 up, 8 in 2024-03-11T20:57:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:57:40.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:57:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:57:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:40 smithi157 ceph-mon[33359]: pgmap v392: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-11T20:57:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:40 smithi157 ceph-mon[33359]: osdmap e292: 8 total, 8 up, 8 in 2024-03-11T20:57:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:57:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:40 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi129-82567-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi129-82567-56"}]': finished 2024-03-11T20:57:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_count","val": "2"}]': finished 2024-03-11T20:57:40.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:40 smithi157 ceph-mon[33359]: osdmap e293: 8 total, 8 up, 8 in 2024-03-11T20:57:40.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:57:40.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:57:42.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:41 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_period","val": "600"}]': finished 2024-03-11T20:57:42.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:41 smithi129 ceph-mon[30184]: osdmap e294: 8 total, 8 up, 8 in 2024-03-11T20:57:42.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:41 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T20:57:42.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:41 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T20:57:42.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:41 smithi129 ceph-mon[30184]: pgmap v395: 300 pgs: 8 unknown, 32 creating+peering, 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:42.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:41 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_period","val": "600"}]': finished 2024-03-11T20:57:42.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:41 smithi129 ceph-mon[42066]: osdmap e294: 8 total, 8 up, 8 in 2024-03-11T20:57:42.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:41 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T20:57:42.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:41 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T20:57:42.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:41 smithi129 ceph-mon[42066]: pgmap v395: 300 pgs: 8 unknown, 32 creating+peering, 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:42.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:41 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_period","val": "600"}]': finished 2024-03-11T20:57:42.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:41 smithi157 ceph-mon[33359]: osdmap e294: 8 total, 8 up, 8 in 2024-03-11T20:57:42.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:41 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T20:57:42.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:41 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T20:57:42.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:41 smithi157 ceph-mon[33359]: pgmap v395: 300 pgs: 8 unknown, 32 creating+peering, 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:43.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:42 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T20:57:43.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:42 smithi129 ceph-mon[42066]: osdmap e295: 8 total, 8 up, 8 in 2024-03-11T20:57:43.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:43.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:42 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:43.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:42 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:57:43.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:42 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T20:57:43.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:42 smithi129 ceph-mon[30184]: osdmap e295: 8 total, 8 up, 8 in 2024-03-11T20:57:43.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:43.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:42 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:43.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:42 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:57:43.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:42 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T20:57:43.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:42 smithi157 ceph-mon[33359]: osdmap e295: 8 total, 8 up, 8 in 2024-03-11T20:57:43.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:43.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:42 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:43.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:42 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:57:43.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:57:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:57:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi129-82567-56"}]': finished 2024-03-11T20:57:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[30184]: osdmap e296: 8 total, 8 up, 8 in 2024-03-11T20:57:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:44.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:44.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41"}]: dispatch 2024-03-11T20:57:44.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41"}]: dispatch 2024-03-11T20:57:44.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[30184]: pgmap v398: 292 pgs: 292 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:44.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:44.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:44.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:44.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi129-82567-56"}]': finished 2024-03-11T20:57:44.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[42066]: osdmap e296: 8 total, 8 up, 8 in 2024-03-11T20:57:44.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:44.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:44.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:44.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:44.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41"}]: dispatch 2024-03-11T20:57:44.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41"}]: dispatch 2024-03-11T20:57:44.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[42066]: pgmap v398: 292 pgs: 292 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:44.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:44.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:44.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:43 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi129-82567-56"}]': finished 2024-03-11T20:57:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:43 smithi157 ceph-mon[33359]: osdmap e296: 8 total, 8 up, 8 in 2024-03-11T20:57:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3462742981' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi129-82567-56"}]: dispatch 2024-03-11T20:57:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:44.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41"}]: dispatch 2024-03-11T20:57:44.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41"}]: dispatch 2024-03-11T20:57:44.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:43 smithi157 ceph-mon[33359]: pgmap v398: 292 pgs: 292 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:44.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:43 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:44.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:43 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:44.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:43 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:45.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi129-82567-56"}]': finished 2024-03-11T20:57:45.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41"}]': finished 2024-03-11T20:57:45.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[30184]: osdmap e297: 8 total, 8 up, 8 in 2024-03-11T20:57:45.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:57:45.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:57:45.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:45.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi129-82567-57"}]: dispatch 2024-03-11T20:57:45.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi129-82567-57"}]: dispatch 2024-03-11T20:57:45.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi129-82567-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:45.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi129-82567-56"}]': finished 2024-03-11T20:57:45.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41"}]': finished 2024-03-11T20:57:45.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[42066]: osdmap e297: 8 total, 8 up, 8 in 2024-03-11T20:57:45.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:57:45.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:57:45.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:45.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi129-82567-57"}]: dispatch 2024-03-11T20:57:45.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi129-82567-57"}]: dispatch 2024-03-11T20:57:45.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi129-82567-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:45.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:44 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi129-82567-56"}]': finished 2024-03-11T20:57:45.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:44 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-41"}]': finished 2024-03-11T20:57:45.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:44 smithi157 ceph-mon[33359]: osdmap e297: 8 total, 8 up, 8 in 2024-03-11T20:57:45.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:44 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:57:45.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:44 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:57:45.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:44 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:45.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi129-82567-57"}]: dispatch 2024-03-11T20:57:45.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi129-82567-57"}]: dispatch 2024-03-11T20:57:45.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi129-82567-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:46.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi129-82567-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:46.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:45 smithi129 ceph-mon[30184]: osdmap e298: 8 total, 8 up, 8 in 2024-03-11T20:57:46.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi129-82567-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi129-82567-57"}]: dispatch 2024-03-11T20:57:46.105 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:45 smithi129 ceph-mon[30184]: pgmap v401: 260 pgs: 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:46.105 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi129-82567-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:46.105 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:45 smithi129 ceph-mon[42066]: osdmap e298: 8 total, 8 up, 8 in 2024-03-11T20:57:46.105 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi129-82567-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi129-82567-57"}]: dispatch 2024-03-11T20:57:46.105 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:45 smithi129 ceph-mon[42066]: pgmap v401: 260 pgs: 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:46.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi129-82567-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:46.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:45 smithi157 ceph-mon[33359]: osdmap e298: 8 total, 8 up, 8 in 2024-03-11T20:57:46.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi129-82567-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi129-82567-57"}]: dispatch 2024-03-11T20:57:46.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:45 smithi157 ceph-mon[33359]: pgmap v401: 260 pgs: 260 active+clean; 8.4 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:46.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:57:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:57:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:57:47.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:46 smithi129 ceph-mon[42066]: osdmap e299: 8 total, 8 up, 8 in 2024-03-11T20:57:47.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:46 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:47.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:46 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:47.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:46 smithi129 ceph-mon[30184]: osdmap e299: 8 total, 8 up, 8 in 2024-03-11T20:57:47.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:47.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:46 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:47.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:46 smithi157 ceph-mon[33359]: osdmap e299: 8 total, 8 up, 8 in 2024-03-11T20:57:47.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:46 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:47.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:46 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:48.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi129-82567-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi129-82567-57"}]': finished 2024-03-11T20:57:48.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:47 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:48.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:47 smithi129 ceph-mon[42066]: osdmap e300: 8 total, 8 up, 8 in 2024-03-11T20:57:48.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:47 smithi129 ceph-mon[42066]: pgmap v404: 300 pgs: 1 creating+activating, 31 creating+peering, 8 unknown, 260 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:48.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi129-82818-6","var": "pg_num","format": "json"}]: dispatch 2024-03-11T20:57:48.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:48.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:47 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:48.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi129-82567-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi129-82567-57"}]': finished 2024-03-11T20:57:48.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:47 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:48.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:47 smithi129 ceph-mon[30184]: osdmap e300: 8 total, 8 up, 8 in 2024-03-11T20:57:48.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:47 smithi129 ceph-mon[30184]: pgmap v404: 300 pgs: 1 creating+activating, 31 creating+peering, 8 unknown, 260 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:48.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi129-82818-6","var": "pg_num","format": "json"}]: dispatch 2024-03-11T20:57:48.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:48.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:47 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:48.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi129-82567-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi129-82567-57"}]': finished 2024-03-11T20:57:48.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:47 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:48.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:47 smithi157 ceph-mon[33359]: osdmap e300: 8 total, 8 up, 8 in 2024-03-11T20:57:48.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:47 smithi157 ceph-mon[33359]: pgmap v404: 300 pgs: 1 creating+activating, 31 creating+peering, 8 unknown, 260 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:48.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi129-82818-6","var": "pg_num","format": "json"}]: dispatch 2024-03-11T20:57:48.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:48.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:47 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:48 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:48 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:48 smithi129 ceph-mon[30184]: osdmap e301: 8 total, 8 up, 8 in 2024-03-11T20:57:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-11T20:57:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:48 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-11T20:57:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:48 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:48 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:48 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_count","val": "8"}]': finished 2024-03-11T20:57:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:48 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:48 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:48 smithi129 ceph-mon[42066]: osdmap e301: 8 total, 8 up, 8 in 2024-03-11T20:57:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-11T20:57:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:48 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-11T20:57:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:48 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:48 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:48 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_count","val": "8"}]': finished 2024-03-11T20:57:49.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:48 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:49.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:48 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:49.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:48 smithi157 ceph-mon[33359]: osdmap e301: 8 total, 8 up, 8 in 2024-03-11T20:57:49.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-11T20:57:49.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:48 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-11T20:57:49.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:48 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:57:49.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:48 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:57:49.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:48 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_count","val": "8"}]': finished 2024-03-11T20:57:50.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[30184]: osdmap e302: 8 total, 8 up, 8 in 2024-03-11T20:57:50.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[30184]: pgmap v407: 292 pgs: 1 creating+activating, 31 creating+peering, 260 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:50.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:57:50.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:57:50.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi129-82567-57"}]: dispatch 2024-03-11T20:57:50.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_period","val": "600"}]': finished 2024-03-11T20:57:50.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi129-82567-57"}]': finished 2024-03-11T20:57:50.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[30184]: osdmap e303: 8 total, 8 up, 8 in 2024-03-11T20:57:50.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi129-82567-57"}]: dispatch 2024-03-11T20:57:50.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[42066]: osdmap e302: 8 total, 8 up, 8 in 2024-03-11T20:57:50.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[42066]: pgmap v407: 292 pgs: 1 creating+activating, 31 creating+peering, 260 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:50.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:57:50.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:57:50.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi129-82567-57"}]: dispatch 2024-03-11T20:57:50.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_period","val": "600"}]': finished 2024-03-11T20:57:50.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi129-82567-57"}]': finished 2024-03-11T20:57:50.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[42066]: osdmap e303: 8 total, 8 up, 8 in 2024-03-11T20:57:50.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi129-82567-57"}]: dispatch 2024-03-11T20:57:50.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:49 smithi157 ceph-mon[33359]: osdmap e302: 8 total, 8 up, 8 in 2024-03-11T20:57:50.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:49 smithi157 ceph-mon[33359]: pgmap v407: 292 pgs: 1 creating+activating, 31 creating+peering, 260 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:50.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:57:50.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:57:50.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi129-82567-57"}]: dispatch 2024-03-11T20:57:50.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_period","val": "600"}]': finished 2024-03-11T20:57:50.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi129-82567-57"}]': finished 2024-03-11T20:57:50.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:49 smithi157 ceph-mon[33359]: osdmap e303: 8 total, 8 up, 8 in 2024-03-11T20:57:50.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi129-82567-57"}]: dispatch 2024-03-11T20:57:51.059 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc:Read 2024-03-11T20:57:51.059 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-11T20:57:51.059 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: ok, hit_set contains 274:602f83fe:::foo:head 2024-03-11T20:57:51.059 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9063 ms) 2024-03-11T20:57:51.059 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-03-11T20:57:51.059 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg_num = 32 2024-03-11T20:57:51.059 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 0 ls 1710190671,0 2024-03-11T20:57:51.059 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 1 ls 1710190671,0 2024-03-11T20:57:51.059 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 2 ls 1710190671,0 2024-03-11T20:57:51.059 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 3 ls 1710190671,0 2024-03-11T20:57:51.059 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 4 ls 1710190671,0 2024-03-11T20:57:51.060 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 5 ls 1710190671,0 2024-03-11T20:57:51.060 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 6 ls 1710190671,0 2024-03-11T20:57:51.060 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 7 ls 1710190671,0 2024-03-11T20:57:51.060 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 8 ls 1710190671,0 2024-03-11T20:57:51.060 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 9 ls 1710190671,0 2024-03-11T20:57:51.060 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 10 ls 1710190671,0 2024-03-11T20:57:51.060 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 11 ls 1710190671,0 2024-03-11T20:57:51.060 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 12 ls 1710190671,0 2024-03-11T20:57:51.060 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 13 ls 1710190671,0 2024-03-11T20:57:51.060 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 14 ls 1710190671,0 2024-03-11T20:57:51.060 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 15 ls 1710190671,0 2024-03-11T20:57:51.060 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 16 ls 1710190671,0 2024-03-11T20:57:51.060 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 17 ls 1710190671,0 2024-03-11T20:57:51.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 18 ls 1710190671,0 2024-03-11T20:57:51.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 19 ls 1710190671,0 2024-03-11T20:57:51.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 20 ls 1710190671,0 2024-03-11T20:57:51.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 21 ls 1710190671,0 2024-03-11T20:57:51.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 22 ls 1710190671,0 2024-03-11T20:57:51.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 23 ls 1710190671,0 2024-03-11T20:57:51.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 24 ls 1710190671,0 2024-03-11T20:57:51.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 25 ls 1710190671,0 2024-03-11T20:57:51.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 26 ls 1710190671,0 2024-03-11T20:57:51.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 27 ls 1710190671,0 2024-03-11T20:57:51.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 28 ls 1710190671,0 2024-03-11T20:57:51.061 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 29 ls 1710190671,0 2024-03-11T20:57:51.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 30 ls 1710190671,0 2024-03-11T20:57:51.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg 31 ls 1710190671,0 2024-03-11T20:57:51.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: pg_num = 32 2024-03-11T20:57:51.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:6cac518f:::0:head 2024-03-11T20:57:51.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:02547ec2:::1:head 2024-03-11T20:57:51.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:f905c69b:::2:head 2024-03-11T20:57:51.062 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:cfc208b3:::3:head 2024-03-11T20:57:51.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:d83876eb:::4:head 2024-03-11T20:57:51.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:b29083e3:::5:head 2024-03-11T20:57:51.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:c4fdafeb:::6:head 2024-03-11T20:57:51.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:5c6b0b28:::7:head 2024-03-11T20:57:51.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:bd63b0f1:::8:head 2024-03-11T20:57:51.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:e960b815:::9:head 2024-03-11T20:57:51.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:52ea6a34:::10:head 2024-03-11T20:57:51.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:89d3ae78:::11:head 2024-03-11T20:57:51.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:de5d7c5f:::12:head 2024-03-11T20:57:51.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:566253c9:::13:head 2024-03-11T20:57:51.063 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:62a1935d:::14:head 2024-03-11T20:57:51.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:863748b0:::15:head 2024-03-11T20:57:51.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:3958e169:::16:head 2024-03-11T20:57:51.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:4d4dabf9:::17:head 2024-03-11T20:57:51.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:8391935d:::18:head 2024-03-11T20:57:51.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:28883081:::19:head 2024-03-11T20:57:51.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:69259c59:::20:head 2024-03-11T20:57:51.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:4bdb80b7:::21:head 2024-03-11T20:57:51.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:a11c5d71:::22:head 2024-03-11T20:57:51.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:271af37b:::23:head 2024-03-11T20:57:51.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:95b121be:::24:head 2024-03-11T20:57:51.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:58d1031b:::25:head 2024-03-11T20:57:51.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:0a050783:::26:head 2024-03-11T20:57:51.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:c709704c:::27:head 2024-03-11T20:57:51.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:cbe56eaf:::28:head 2024-03-11T20:57:51.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:86b4b162:::29:head 2024-03-11T20:57:51.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:70d89383:::30:head 2024-03-11T20:57:51.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:dd450c7c:::31:head 2024-03-11T20:57:51.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:6d5729b1:::32:head 2024-03-11T20:57:51.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:c388f3fb:::33:head 2024-03-11T20:57:51.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:56cfea31:::34:head 2024-03-11T20:57:51.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:9dbc1bf7:::35:head 2024-03-11T20:57:51.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:40b74ccd:::36:head 2024-03-11T20:57:51.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:4d5aaf42:::37:head 2024-03-11T20:57:51.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:920f362c:::38:head 2024-03-11T20:57:51.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:6cc53222:::39:head 2024-03-11T20:57:51.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:9cad833f:::40:head 2024-03-11T20:57:51.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:1ea84d41:::41:head 2024-03-11T20:57:51.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:c4480ef6:::42:head 2024-03-11T20:57:51.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:a694361e:::43:head 2024-03-11T20:57:51.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:d1bd33e9:::44:head 2024-03-11T20:57:51.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:ddc2cd5d:::45:head 2024-03-11T20:57:51.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:2b782207:::46:head 2024-03-11T20:57:51.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:7b187fca:::47:head 2024-03-11T20:57:51.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:90ecdf6f:::48:head 2024-03-11T20:57:51.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:a5ed95fe:::49:head 2024-03-11T20:57:51.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:ea0eaa55:::50:head 2024-03-11T20:57:51.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:f33ef17b:::51:head 2024-03-11T20:57:51.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:a0d1b2f6:::52:head 2024-03-11T20:57:51.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:60c5229e:::53:head 2024-03-11T20:57:51.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:edcbc575:::54:head 2024-03-11T20:57:51.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:102cf253:::55:head 2024-03-11T20:57:51.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:efb7fb0b:::56:head 2024-03-11T20:57:51.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:50d0a326:::57:head 2024-03-11T20:57:51.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:d4dc5daf:::58:head 2024-03-11T20:57:51.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:3a130462:::59:head 2024-03-11T20:57:51.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:ec87ed71:::60:head 2024-03-11T20:57:51.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:d5bc9454:::61:head 2024-03-11T20:57:51.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:3ddfe313:::62:head 2024-03-11T20:57:51.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:7c2816b9:::63:head 2024-03-11T20:57:51.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:47e00e4d:::64:head 2024-03-11T20:57:51.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:c6410c18:::65:head 2024-03-11T20:57:51.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:b48ed237:::66:head 2024-03-11T20:57:51.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:cd63ad31:::67:head 2024-03-11T20:57:51.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:b179e92b:::68:head 2024-03-11T20:57:51.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:0d9f741a:::69:head 2024-03-11T20:57:51.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:6d3352ae:::70:head 2024-03-11T20:57:51.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:c6d5c19e:::71:head 2024-03-11T20:57:51.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:bc4729c3:::72:head 2024-03-11T20:57:51.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:77e930b9:::73:head 2024-03-11T20:57:51.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:0abeecfd:::74:head 2024-03-11T20:57:51.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:b7c37e15:::75:head 2024-03-11T20:57:51.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:b6378398:::76:head 2024-03-11T20:57:51.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:02bd68de:::77:head 2024-03-11T20:57:51.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:cc795d2d:::78:head 2024-03-11T20:57:51.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:630d4fea:::79:head 2024-03-11T20:57:51.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:e0d29ef5:::80:head 2024-03-11T20:57:51.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:fd6f13d2:::81:head 2024-03-11T20:57:51.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:606461d5:::82:head 2024-03-11T20:57:51.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:eadbdc43:::83:head 2024-03-11T20:57:51.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:8761d0bb:::84:head 2024-03-11T20:57:51.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:9ef0186f:::85:head 2024-03-11T20:57:51.069 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:e0d41294:::86:head 2024-03-11T20:57:51.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:961de695:::87:head 2024-03-11T20:57:51.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:1423148f:::88:head 2024-03-11T20:57:51.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:633a8fa2:::89:head 2024-03-11T20:57:51.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:a8653809:::90:head 2024-03-11T20:57:51.070 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:3dac8b33:::91:head 2024-03-11T20:57:51.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:35aad435:::92:head 2024-03-11T20:57:51.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:f6dcc343:::93:head 2024-03-11T20:57:51.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:dbbdad87:::94:head 2024-03-11T20:57:51.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:1cb48ce0:::95:head 2024-03-11T20:57:51.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:03cd461c:::96:head 2024-03-11T20:57:51.071 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:17a4ea99:::97:head 2024-03-11T20:57:51.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:9993c9a7:::98:head 2024-03-11T20:57:51.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:6394211c:::99:head 2024-03-11T20:57:51.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:94c7ae57:::100:head 2024-03-11T20:57:51.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:6fdee5bb:::101:head 2024-03-11T20:57:51.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:9a477fd1:::102:head 2024-03-11T20:57:51.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:eb850916:::103:head 2024-03-11T20:57:51.072 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:affc56b9:::104:head 2024-03-11T20:57:51.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:b42dc814:::105:head 2024-03-11T20:57:51.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:f319f8f0:::106:head 2024-03-11T20:57:51.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:9a40b9de:::107:head 2024-03-11T20:57:51.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:8b524f28:::108:head 2024-03-11T20:57:51.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:e3de589f:::109:head 2024-03-11T20:57:51.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:90f90a5b:::110:head 2024-03-11T20:57:51.073 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:a7b4f1d7:::111:head 2024-03-11T20:57:51.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:af51766e:::112:head 2024-03-11T20:57:51.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:b6f90bd1:::113:head 2024-03-11T20:57:51.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:e0261208:::114:head 2024-03-11T20:57:51.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:c9569ef7:::115:head 2024-03-11T20:57:51.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:61bebe50:::116:head 2024-03-11T20:57:51.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:fe93412b:::117:head 2024-03-11T20:57:51.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:d3d38bee:::118:head 2024-03-11T20:57:51.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:3100ba0c:::119:head 2024-03-11T20:57:51.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:d0560ada:::120:head 2024-03-11T20:57:51.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:f0ea8b35:::121:head 2024-03-11T20:57:51.074 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:766f231a:::122:head 2024-03-11T20:57:51.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:a07a2582:::123:head 2024-03-11T20:57:51.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:bd7c6b3a:::124:head 2024-03-11T20:57:51.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:fb2ddaff:::125:head 2024-03-11T20:57:51.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:4408e1fe:::126:head 2024-03-11T20:57:51.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:ee1df7a7:::127:head 2024-03-11T20:57:51.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:c3002909:::128:head 2024-03-11T20:57:51.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:4f48ffa9:::129:head 2024-03-11T20:57:51.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:edf38733:::130:head 2024-03-11T20:57:51.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:c08425c0:::131:head 2024-03-11T20:57:51.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:5f902d98:::132:head 2024-03-11T20:57:51.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:41ea2c93:::133:head 2024-03-11T20:57:51.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:813cee13:::134:head 2024-03-11T20:57:51.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:0131818d:::135:head 2024-03-11T20:57:51.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:26ba5a85:::136:head 2024-03-11T20:57:51.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:381b8a5a:::137:head 2024-03-11T20:57:51.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:28797e47:::138:head 2024-03-11T20:57:51.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:bfca7f22:::139:head 2024-03-11T20:57:51.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:36807075:::140:head 2024-03-11T20:57:51.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:80b03975:::141:head 2024-03-11T20:57:51.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:5c15709b:::142:head 2024-03-11T20:57:51.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:f39ea15e:::143:head 2024-03-11T20:57:51.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:ea992956:::144:head 2024-03-11T20:57:51.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:48887b1c:::145:head 2024-03-11T20:57:51.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:9f24a9dd:::146:head 2024-03-11T20:57:51.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:987f100b:::147:head 2024-03-11T20:57:51.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:d2dd3581:::148:head 2024-03-11T20:57:51.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:7fed1808:::149:head 2024-03-11T20:57:51.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:c80b70e9:::150:head 2024-03-11T20:57:51.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:85ed90f9:::151:head 2024-03-11T20:57:51.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:36428b24:::152:head 2024-03-11T20:57:51.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:d044c34a:::153:head 2024-03-11T20:57:51.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:7c18bf58:::154:head 2024-03-11T20:57:51.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:d1c21232:::155:head 2024-03-11T20:57:51.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:a7a3c575:::156:head 2024-03-11T20:57:51.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:87da0633:::157:head 2024-03-11T20:57:51.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:d5ac3822:::158:head 2024-03-11T20:57:51.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:3f20522d:::159:head 2024-03-11T20:57:51.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:6ca26563:::160:head 2024-03-11T20:57:51.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:532ce135:::161:head 2024-03-11T20:57:51.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:c78863e6:::162:head 2024-03-11T20:57:51.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-11T20:57:51.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-11T20:57:51.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi129-82567-57"}]': finished 2024-03-11T20:57:51.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-11T20:57:51.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:50 smithi129 ceph-mon[30184]: osdmap e304: 8 total, 8 up, 8 in 2024-03-11T20:57:51.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-11T20:57:51.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-11T20:57:51.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi129-82567-57"}]': finished 2024-03-11T20:57:51.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-11T20:57:51.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:50 smithi129 ceph-mon[42066]: osdmap e304: 8 total, 8 up, 8 in 2024-03-11T20:57:51.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-11T20:57:51.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-11T20:57:51.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2725824403' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi129-82567-57"}]': finished 2024-03-11T20:57:51.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-11T20:57:51.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:50 smithi157 ceph-mon[33359]: osdmap e304: 8 total, 8 up, 8 in 2024-03-11T20:57:52.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[42066]: pgmap v410: 292 pgs: 1 creating+activating, 31 creating+peering, 260 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:52.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:52.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:52.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:52.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:52.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi129-82567-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:52.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi129-82567-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:52.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi129-82818-43","var": "pg_num","format": "json"}]: dispatch 2024-03-11T20:57:52.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:52.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:52.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43"}]: dispatch 2024-03-11T20:57:52.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43"}]: dispatch 2024-03-11T20:57:52.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[30184]: pgmap v410: 292 pgs: 1 creating+activating, 31 creating+peering, 260 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:52.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:52.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:52.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:52.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:52.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi129-82567-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:52.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi129-82567-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:52.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi129-82818-43","var": "pg_num","format": "json"}]: dispatch 2024-03-11T20:57:52.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:52.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:52.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43"}]: dispatch 2024-03-11T20:57:52.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:51 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43"}]: dispatch 2024-03-11T20:57:52.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:51 smithi157 ceph-mon[33359]: pgmap v410: 292 pgs: 1 creating+activating, 31 creating+peering, 260 active+clean; 8.4 MiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:57:52.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:52.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:51 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:52.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:52.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:51 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:52.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi129-82567-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:52.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:51 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi129-82567-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:52.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi129-82818-43","var": "pg_num","format": "json"}]: dispatch 2024-03-11T20:57:52.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:52.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:51 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:57:52.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43"}]: dispatch 2024-03-11T20:57:52.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:51 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43"}]: dispatch 2024-03-11T20:57:53.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi129-82567-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:53.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43"}]': finished 2024-03-11T20:57:53.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:52 smithi129 ceph-mon[42066]: osdmap e305: 8 total, 8 up, 8 in 2024-03-11T20:57:53.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi129-82567-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:53.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi129-82567-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:53.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi129-82567-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:53.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43"}]': finished 2024-03-11T20:57:53.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:52 smithi129 ceph-mon[30184]: osdmap e305: 8 total, 8 up, 8 in 2024-03-11T20:57:53.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi129-82567-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:53.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi129-82567-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:53.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi129-82567-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:53.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-43"}]': finished 2024-03-11T20:57:53.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:52 smithi157 ceph-mon[33359]: osdmap e305: 8 total, 8 up, 8 in 2024-03-11T20:57:53.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi129-82567-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:53.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi129-82567-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:53.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:57:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:57:54.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:53 smithi129 ceph-mon[30184]: pgmap v412: 292 pgs: 292 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 50 KiB/s wr, 50 op/s 2024-03-11T20:57:54.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:53 smithi129 ceph-mon[30184]: osdmap e306: 8 total, 8 up, 8 in 2024-03-11T20:57:54.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:53 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:54.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:53 smithi129 ceph-mon[42066]: pgmap v412: 292 pgs: 292 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 50 KiB/s wr, 50 op/s 2024-03-11T20:57:54.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:53 smithi129 ceph-mon[42066]: osdmap e306: 8 total, 8 up, 8 in 2024-03-11T20:57:54.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:53 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:54.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:53 smithi157 ceph-mon[33359]: pgmap v412: 292 pgs: 292 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 50 KiB/s wr, 50 op/s 2024-03-11T20:57:54.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:53 smithi157 ceph-mon[33359]: osdmap e306: 8 total, 8 up, 8 in 2024-03-11T20:57:54.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:53 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:57:55.130 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi129-82567-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi129-82567-58"}]': finished 2024-03-11T20:57:55.130 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:54 smithi129 ceph-mon[30184]: osdmap e307: 8 total, 8 up, 8 in 2024-03-11T20:57:55.131 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:55.131 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:55.131 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:54 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:55.131 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi129-82567-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi129-82567-58"}]': finished 2024-03-11T20:57:55.131 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:54 smithi129 ceph-mon[42066]: osdmap e307: 8 total, 8 up, 8 in 2024-03-11T20:57:55.131 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:55.131 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:55.131 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:54 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:55.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi129-82567-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi129-82567-58"}]': finished 2024-03-11T20:57:55.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:54 smithi157 ceph-mon[33359]: osdmap e307: 8 total, 8 up, 8 in 2024-03-11T20:57:55.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:55.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:57:55.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:54 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:57:56.229 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:57:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:57:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:57:56.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:55 smithi129 ceph-mon[30184]: pgmap v415: 300 pgs: 40 unknown, 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:57:56.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:56.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:55 smithi129 ceph-mon[30184]: osdmap e308: 8 total, 8 up, 8 in 2024-03-11T20:57:56.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:56.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:56.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:55 smithi129 ceph-mon[42066]: pgmap v415: 300 pgs: 40 unknown, 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:57:56.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:56.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:55 smithi129 ceph-mon[42066]: osdmap e308: 8 total, 8 up, 8 in 2024-03-11T20:57:56.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:56.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:56.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:55 smithi157 ceph-mon[33359]: pgmap v415: 300 pgs: 40 unknown, 260 active+clean; 8.4 MiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T20:57:56.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:57:56.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:55 smithi157 ceph-mon[33359]: osdmap e308: 8 total, 8 up, 8 in 2024-03-11T20:57:56.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:56.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:57:57.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:56 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:57.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:56 smithi129 ceph-mon[30184]: osdmap e309: 8 total, 8 up, 8 in 2024-03-11T20:57:57.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T20:57:57.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:57.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:56 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T20:57:57.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:56 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:57.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:56 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:57.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:56 smithi129 ceph-mon[42066]: osdmap e309: 8 total, 8 up, 8 in 2024-03-11T20:57:57.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T20:57:57.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:57.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:56 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T20:57:57.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:56 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:57.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:56 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:57:57.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:56 smithi157 ceph-mon[33359]: osdmap e309: 8 total, 8 up, 8 in 2024-03-11T20:57:57.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T20:57:57.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:57.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:56 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T20:57:57.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:56 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:58.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:57 smithi129 ceph-mon[30184]: pgmap v418: 292 pgs: 292 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:58.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_count","val": "3"}]': finished 2024-03-11T20:57:58.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi129-82567-58"}]': finished 2024-03-11T20:57:58.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:57 smithi129 ceph-mon[30184]: osdmap e310: 8 total, 8 up, 8 in 2024-03-11T20:57:58.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:58.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T20:57:58.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:58.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T20:57:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:57 smithi129 ceph-mon[42066]: pgmap v418: 292 pgs: 292 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_count","val": "3"}]': finished 2024-03-11T20:57:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi129-82567-58"}]': finished 2024-03-11T20:57:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:57 smithi129 ceph-mon[42066]: osdmap e310: 8 total, 8 up, 8 in 2024-03-11T20:57:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:58.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T20:57:58.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:58.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T20:57:58.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:57 smithi157 ceph-mon[33359]: pgmap v418: 292 pgs: 292 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:57:58.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_count","val": "3"}]': finished 2024-03-11T20:57:58.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi129-82567-58"}]': finished 2024-03-11T20:57:58.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:57 smithi157 ceph-mon[33359]: osdmap e310: 8 total, 8 up, 8 in 2024-03-11T20:57:58.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3174616255' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:58.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T20:57:58.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi129-82567-58"}]: dispatch 2024-03-11T20:57:58.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T20:57:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi129-82567-58"}]': finished 2024-03-11T20:57:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_period","val": "3"}]': finished 2024-03-11T20:57:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[30184]: osdmap e311: 8 total, 8 up, 8 in 2024-03-11T20:57:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:57:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:57:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi129-82567-59"}]: dispatch 2024-03-11T20:57:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi129-82567-59"}]: dispatch 2024-03-11T20:57:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi129-82567-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:59.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T20:57:59.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi129-82567-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:59.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[30184]: osdmap e312: 8 total, 8 up, 8 in 2024-03-11T20:57:59.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi129-82567-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi129-82567-59"}]: dispatch 2024-03-11T20:57:59.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T20:57:59.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T20:57:59.234 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi129-82567-58"}]': finished 2024-03-11T20:57:59.234 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_period","val": "3"}]': finished 2024-03-11T20:57:59.234 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[42066]: osdmap e311: 8 total, 8 up, 8 in 2024-03-11T20:57:59.234 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:57:59.234 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:57:59.234 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi129-82567-59"}]: dispatch 2024-03-11T20:57:59.234 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi129-82567-59"}]: dispatch 2024-03-11T20:57:59.234 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi129-82567-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:59.234 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T20:57:59.235 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi129-82567-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:59.235 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[42066]: osdmap e312: 8 total, 8 up, 8 in 2024-03-11T20:57:59.235 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi129-82567-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi129-82567-59"}]: dispatch 2024-03-11T20:57:59.235 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T20:57:59.235 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:57:58 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T20:57:59.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:58 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi129-82567-58"}]': finished 2024-03-11T20:57:59.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:58 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_period","val": "3"}]': finished 2024-03-11T20:57:59.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:58 smithi157 ceph-mon[33359]: osdmap e311: 8 total, 8 up, 8 in 2024-03-11T20:57:59.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:57:59.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:58 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:57:59.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi129-82567-59"}]: dispatch 2024-03-11T20:57:59.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi129-82567-59"}]: dispatch 2024-03-11T20:57:59.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi129-82567-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:57:59.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:58 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T20:57:59.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi129-82567-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:57:59.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:58 smithi157 ceph-mon[33359]: osdmap e312: 8 total, 8 up, 8 in 2024-03-11T20:57:59.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi129-82567-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi129-82567-59"}]: dispatch 2024-03-11T20:57:59.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T20:57:59.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:57:58 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T20:58:00.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:00 smithi129 ceph-mon[30184]: pgmap v421: 292 pgs: 292 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:00.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:00 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-11T20:58:00.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:00 smithi129 ceph-mon[42066]: pgmap v421: 292 pgs: 292 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:00.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:00 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-11T20:58:00.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:00 smithi157 ceph-mon[33359]: pgmap v421: 292 pgs: 292 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:00.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:00 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-11T20:58:01.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:01 smithi129 ceph-mon[30184]: osdmap e313: 8 total, 8 up, 8 in 2024-03-11T20:58:01.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi129-82567-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi129-82567-59"}]': finished 2024-03-11T20:58:01.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:01 smithi129 ceph-mon[30184]: osdmap e314: 8 total, 8 up, 8 in 2024-03-11T20:58:01.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:01 smithi129 ceph-mon[42066]: osdmap e313: 8 total, 8 up, 8 in 2024-03-11T20:58:01.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi129-82567-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi129-82567-59"}]': finished 2024-03-11T20:58:01.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:01 smithi129 ceph-mon[42066]: osdmap e314: 8 total, 8 up, 8 in 2024-03-11T20:58:01.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:01 smithi157 ceph-mon[33359]: osdmap e313: 8 total, 8 up, 8 in 2024-03-11T20:58:01.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi129-82567-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi129-82567-59"}]': finished 2024-03-11T20:58:01.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:01 smithi157 ceph-mon[33359]: osdmap e314: 8 total, 8 up, 8 in 2024-03-11T20:58:02.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:02 smithi129 ceph-mon[30184]: pgmap v424: 292 pgs: 292 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:02.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:02 smithi129 ceph-mon[30184]: osdmap e315: 8 total, 8 up, 8 in 2024-03-11T20:58:02.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:02 smithi129 ceph-mon[42066]: pgmap v424: 292 pgs: 292 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:02.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:02 smithi129 ceph-mon[42066]: osdmap e315: 8 total, 8 up, 8 in 2024-03-11T20:58:02.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:02 smithi157 ceph-mon[33359]: pgmap v424: 292 pgs: 292 active+clean; 8.4 MiB data, 602 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:02.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:02 smithi157 ceph-mon[33359]: osdmap e315: 8 total, 8 up, 8 in 2024-03-11T20:58:03.901 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:58:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:58:04.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:03 smithi129 ceph-mon[30184]: pgmap v427: 300 pgs: 8 creating+peering, 292 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:58:04.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:03 smithi129 ceph-mon[30184]: osdmap e316: 8 total, 8 up, 8 in 2024-03-11T20:58:04.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi129-82567-59"}]: dispatch 2024-03-11T20:58:04.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:03 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:04.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:03 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:04.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:03 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:04.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:03 smithi129 ceph-mon[42066]: pgmap v427: 300 pgs: 8 creating+peering, 292 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:58:04.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:03 smithi129 ceph-mon[42066]: osdmap e316: 8 total, 8 up, 8 in 2024-03-11T20:58:04.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi129-82567-59"}]: dispatch 2024-03-11T20:58:04.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:03 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:04.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:03 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:04.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:03 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:04.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:03 smithi157 ceph-mon[33359]: pgmap v427: 300 pgs: 8 creating+peering, 292 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:58:04.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:03 smithi157 ceph-mon[33359]: osdmap e316: 8 total, 8 up, 8 in 2024-03-11T20:58:04.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi129-82567-59"}]: dispatch 2024-03-11T20:58:04.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:03 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:04.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:03 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:04.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:03 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:05.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi129-82567-59"}]': finished 2024-03-11T20:58:05.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:04 smithi129 ceph-mon[30184]: osdmap e317: 8 total, 8 up, 8 in 2024-03-11T20:58:05.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi129-82567-59"}]: dispatch 2024-03-11T20:58:05.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi129-82567-59"}]': finished 2024-03-11T20:58:05.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi129-82567-59"}]': finished 2024-03-11T20:58:05.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:04 smithi129 ceph-mon[42066]: osdmap e317: 8 total, 8 up, 8 in 2024-03-11T20:58:05.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi129-82567-59"}]: dispatch 2024-03-11T20:58:05.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi129-82567-59"}]': finished 2024-03-11T20:58:05.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi129-82567-59"}]': finished 2024-03-11T20:58:05.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:04 smithi157 ceph-mon[33359]: osdmap e317: 8 total, 8 up, 8 in 2024-03-11T20:58:05.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi129-82567-59"}]: dispatch 2024-03-11T20:58:05.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/478658308' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi129-82567-59"}]': finished 2024-03-11T20:58:06.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:05 smithi129 ceph-mon[30184]: pgmap v430: 292 pgs: 292 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:58:06.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:05 smithi129 ceph-mon[30184]: osdmap e318: 8 total, 8 up, 8 in 2024-03-11T20:58:06.229 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:58:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:58:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:58:06.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:05 smithi129 ceph-mon[42066]: pgmap v430: 292 pgs: 292 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:58:06.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:05 smithi129 ceph-mon[42066]: osdmap e318: 8 total, 8 up, 8 in 2024-03-11T20:58:06.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:05 smithi157 ceph-mon[33359]: pgmap v430: 292 pgs: 292 active+clean; 8.4 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:58:06.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:05 smithi157 ceph-mon[33359]: osdmap e318: 8 total, 8 up, 8 in 2024-03-11T20:58:07.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:06 smithi129 ceph-mon[30184]: osdmap e319: 8 total, 8 up, 8 in 2024-03-11T20:58:07.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3870605283' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi129-82567-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:07.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:06 smithi129 ceph-mon[42066]: osdmap e319: 8 total, 8 up, 8 in 2024-03-11T20:58:07.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3870605283' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi129-82567-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:07.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:06 smithi157 ceph-mon[33359]: osdmap e319: 8 total, 8 up, 8 in 2024-03-11T20:58:07.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3870605283' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi129-82567-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:07.938 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:7462ddf6:::1.RoundTripAppendPP (3234 ms) 2024-03-11T20:58:07.938 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-03-11T20:58:07.938 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2993 ms) 2024-03-11T20:58:07.938 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-03-11T20:58:07.939 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2783 ms) 2024-03-11T20:58:07.939 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-03-11T20:58:07.939 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3005 ms) 2024-03-11T20:58:07.939 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-03-11T20:58:07.939 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: setting pool EIO 2024-03-11T20:58:07.939 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: max_success 99, min_failed 100 2024-03-11T20:58:07.939 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4021 ms) 2024-03-11T20:58:07.939 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-03-11T20:58:07.939 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3027 ms) 2024-03-11T20:58:07.939 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (124087 ms total) 2024-03-11T20:58:07.939 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: 2024-03-11T20:58:07.939 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-03-11T20:58:07.939 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-03-11T20:58:07.940 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3007 ms) 2024-03-11T20:58:07.940 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-03-11T20:58:07.940 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9064 ms) 2024-03-11T20:58:07.940 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-03-11T20:58:07.940 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14889 ms) 2024-03-11T20:58:07.940 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-03-11T20:58:07.940 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3010 ms) 2024-03-11T20:58:07.940 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29970 ms total) 2024-03-11T20:58:07.940 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: 2024-03-11T20:58:07.940 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-03-11T20:58:07.940 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-11T20:58:07.940 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3252 ms) 2024-03-11T20:58:07.940 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3252 ms total) 2024-03-11T20:58:07.940 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: 2024-03-11T20:58:07.941 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-03-11T20:58:07.941 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-03-11T20:58:07.941 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13710 ms) 2024-03-11T20:58:07.941 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-03-11T20:58:07.941 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7033 ms) 2024-03-11T20:58:07.941 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-03-11T20:58:07.941 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6984 ms) 2024-03-11T20:58:07.941 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-03-11T20:58:07.941 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7019 ms) 2024-03-11T20:58:07.941 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-03-11T20:58:07.941 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3011 ms) 2024-03-11T20:58:07.941 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-03-11T20:58:07.941 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7028 ms) 2024-03-11T20:58:07.941 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-03-11T20:58:07.941 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7010 ms) 2024-03-11T20:58:07.942 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-03-11T20:58:07.942 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6958 ms) 2024-03-11T20:58:07.942 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-03-11T20:58:07.942 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7032 ms) 2024-03-11T20:58:07.942 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-03-11T20:58:07.942 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7031 ms) 2024-03-11T20:58:07.942 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-03-11T20:58:07.942 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7042 ms) 2024-03-11T20:58:07.942 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-03-11T20:58:07.942 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7062 ms) 2024-03-11T20:58:07.942 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-03-11T20:58:07.942 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7016 ms) 2024-03-11T20:58:07.942 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-03-11T20:58:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:07 smithi129 ceph-mon[30184]: pgmap v433: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-11T20:58:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3870605283' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi129-82567-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:58:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:07 smithi129 ceph-mon[30184]: osdmap e320: 8 total, 8 up, 8 in 2024-03-11T20:58:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:07 smithi129 ceph-mon[42066]: pgmap v433: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-11T20:58:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3870605283' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi129-82567-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:58:08.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:07 smithi129 ceph-mon[42066]: osdmap e320: 8 total, 8 up, 8 in 2024-03-11T20:58:08.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:07 smithi157 ceph-mon[33359]: pgmap v433: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-11T20:58:08.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3870605283' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi129-82567-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:58:08.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:07 smithi157 ceph-mon[33359]: osdmap e320: 8 total, 8 up, 8 in 2024-03-11T20:58:09.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:08 smithi129 ceph-mon[42066]: osdmap e321: 8 total, 8 up, 8 in 2024-03-11T20:58:09.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:09.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:09.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:09.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi129-82567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi129-82567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:09.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:08 smithi129 ceph-mon[30184]: osdmap e321: 8 total, 8 up, 8 in 2024-03-11T20:58:09.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:09.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:09.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:09.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:09.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi129-82567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:09.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi129-82567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:09.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:08 smithi157 ceph-mon[33359]: osdmap e321: 8 total, 8 up, 8 in 2024-03-11T20:58:09.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:09.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:09.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:09.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:09.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi129-82567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:09.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi129-82567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:10.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:10 smithi129 ceph-mon[30184]: pgmap v436: 292 pgs: 292 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-11T20:58:10.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:10 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:10.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi129-82567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:10.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:10 smithi129 ceph-mon[30184]: osdmap e322: 8 total, 8 up, 8 in 2024-03-11T20:58:10.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi129-82567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:10.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi129-82567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:10.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:10 smithi129 ceph-mon[42066]: pgmap v436: 292 pgs: 292 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-11T20:58:10.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:10 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:10.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi129-82567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:10.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:10 smithi129 ceph-mon[42066]: osdmap e322: 8 total, 8 up, 8 in 2024-03-11T20:58:10.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi129-82567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:10.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi129-82567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:10.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:10 smithi157 ceph-mon[33359]: pgmap v436: 292 pgs: 292 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-11T20:58:10.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:10 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:10.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi129-82567-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:10.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:10 smithi157 ceph-mon[33359]: osdmap e322: 8 total, 8 up, 8 in 2024-03-11T20:58:10.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi129-82567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:10.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi129-82567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:11.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:11 smithi129 ceph-mon[30184]: osdmap e323: 8 total, 8 up, 8 in 2024-03-11T20:58:11.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi129-82567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi129-82567-61"}]': finished 2024-03-11T20:58:11.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:11 smithi129 ceph-mon[30184]: osdmap e324: 8 total, 8 up, 8 in 2024-03-11T20:58:11.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:11 smithi129 ceph-mon[42066]: osdmap e323: 8 total, 8 up, 8 in 2024-03-11T20:58:11.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi129-82567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi129-82567-61"}]': finished 2024-03-11T20:58:11.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:11 smithi129 ceph-mon[42066]: osdmap e324: 8 total, 8 up, 8 in 2024-03-11T20:58:11.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:11 smithi157 ceph-mon[33359]: osdmap e323: 8 total, 8 up, 8 in 2024-03-11T20:58:11.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi129-82567-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi129-82567-61"}]': finished 2024-03-11T20:58:11.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:11 smithi157 ceph-mon[33359]: osdmap e324: 8 total, 8 up, 8 in 2024-03-11T20:58:12.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:12 smithi129 ceph-mon[42066]: pgmap v439: 292 pgs: 292 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:12.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:12 smithi129 ceph-mon[42066]: osdmap e325: 8 total, 8 up, 8 in 2024-03-11T20:58:12.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:12.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:12.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45"}]: dispatch 2024-03-11T20:58:12.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45"}]: dispatch 2024-03-11T20:58:12.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:12 smithi129 ceph-mon[30184]: pgmap v439: 292 pgs: 292 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:12.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:12 smithi129 ceph-mon[30184]: osdmap e325: 8 total, 8 up, 8 in 2024-03-11T20:58:12.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:12.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:12.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45"}]: dispatch 2024-03-11T20:58:12.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45"}]: dispatch 2024-03-11T20:58:12.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:12 smithi157 ceph-mon[33359]: pgmap v439: 292 pgs: 292 active+clean; 8.4 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:12.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:12 smithi157 ceph-mon[33359]: osdmap e325: 8 total, 8 up, 8 in 2024-03-11T20:58:12.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:12.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:12.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45"}]: dispatch 2024-03-11T20:58:12.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45"}]: dispatch 2024-03-11T20:58:13.956 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:58:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:58:14.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:13 smithi129 ceph-mon[30184]: pgmap v442: 300 pgs: 8 creating+peering, 292 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-11T20:58:14.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45"}]': finished 2024-03-11T20:58:14.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:13 smithi129 ceph-mon[30184]: osdmap e326: 8 total, 8 up, 8 in 2024-03-11T20:58:14.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:14.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:14.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:13 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:13 smithi129 ceph-mon[42066]: pgmap v442: 300 pgs: 8 creating+peering, 292 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-11T20:58:14.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45"}]': finished 2024-03-11T20:58:14.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:13 smithi129 ceph-mon[42066]: osdmap e326: 8 total, 8 up, 8 in 2024-03-11T20:58:14.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:14.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:14.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:13 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:14.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:13 smithi157 ceph-mon[33359]: pgmap v442: 300 pgs: 8 creating+peering, 292 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-11T20:58:14.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-45"}]': finished 2024-03-11T20:58:14.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:13 smithi157 ceph-mon[33359]: osdmap e326: 8 total, 8 up, 8 in 2024-03-11T20:58:14.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:14.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:14.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:13 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:15.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi129-82567-61"}]': finished 2024-03-11T20:58:15.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:14 smithi129 ceph-mon[42066]: osdmap e327: 8 total, 8 up, 8 in 2024-03-11T20:58:15.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:15.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:15.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:14 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:15.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi129-82567-61"}]': finished 2024-03-11T20:58:15.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:14 smithi129 ceph-mon[42066]: osdmap e328: 8 total, 8 up, 8 in 2024-03-11T20:58:15.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi129-82567-61"}]': finished 2024-03-11T20:58:15.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:14 smithi129 ceph-mon[30184]: osdmap e327: 8 total, 8 up, 8 in 2024-03-11T20:58:15.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:15.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:15.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:14 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:15.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi129-82567-61"}]': finished 2024-03-11T20:58:15.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:14 smithi129 ceph-mon[30184]: osdmap e328: 8 total, 8 up, 8 in 2024-03-11T20:58:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi129-82567-61"}]': finished 2024-03-11T20:58:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:14 smithi157 ceph-mon[33359]: osdmap e327: 8 total, 8 up, 8 in 2024-03-11T20:58:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2252995448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:15.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi129-82567-61"}]: dispatch 2024-03-11T20:58:15.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:14 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:15.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi129-82567-61"}]': finished 2024-03-11T20:58:15.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:14 smithi157 ceph-mon[33359]: osdmap e328: 8 total, 8 up, 8 in 2024-03-11T20:58:16.229 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:58:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:58:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:58:16.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[42066]: pgmap v445: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:16.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:16.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:16.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi129-82567-62"}]: dispatch 2024-03-11T20:58:16.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi129-82567-62"}]: dispatch 2024-03-11T20:58:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi129-82567-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:58:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi129-82567-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[42066]: osdmap e329: 8 total, 8 up, 8 in 2024-03-11T20:58:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi129-82567-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi129-82567-62"}]: dispatch 2024-03-11T20:58:16.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[30184]: pgmap v445: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:16.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:16.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:16.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi129-82567-62"}]: dispatch 2024-03-11T20:58:16.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi129-82567-62"}]: dispatch 2024-03-11T20:58:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi129-82567-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:58:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi129-82567-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[30184]: osdmap e329: 8 total, 8 up, 8 in 2024-03-11T20:58:16.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi129-82567-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi129-82567-62"}]: dispatch 2024-03-11T20:58:16.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:15 smithi157 ceph-mon[33359]: pgmap v445: 260 pgs: 260 active+clean; 8.4 MiB data, 613 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:16.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi129-82567-62"}]: dispatch 2024-03-11T20:58:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi129-82567-62"}]: dispatch 2024-03-11T20:58:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi129-82567-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:58:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi129-82567-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:15 smithi157 ceph-mon[33359]: osdmap e329: 8 total, 8 up, 8 in 2024-03-11T20:58:16.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi129-82567-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi129-82567-62"}]: dispatch 2024-03-11T20:58:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:58:17.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:17.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:17.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:58:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:58:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[30184]: pgmap v448: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T20:58:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[30184]: osdmap e330: 8 total, 8 up, 8 in 2024-03-11T20:58:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi129-82567-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi129-82567-62"}]': finished 2024-03-11T20:58:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-47"}]': finished 2024-03-11T20:58:18.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[30184]: osdmap e331: 8 total, 8 up, 8 in 2024-03-11T20:58:18.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-47", "mode": "writeback"}]: dispatch 2024-03-11T20:58:18.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-47", "mode": "writeback"}]: dispatch 2024-03-11T20:58:18.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[42066]: pgmap v448: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T20:58:18.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:18.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[42066]: osdmap e330: 8 total, 8 up, 8 in 2024-03-11T20:58:18.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:18.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi129-82567-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi129-82567-62"}]': finished 2024-03-11T20:58:18.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-47"}]': finished 2024-03-11T20:58:18.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[42066]: osdmap e331: 8 total, 8 up, 8 in 2024-03-11T20:58:18.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-47", "mode": "writeback"}]: dispatch 2024-03-11T20:58:18.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-47", "mode": "writeback"}]: dispatch 2024-03-11T20:58:18.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:18 smithi157 ceph-mon[33359]: pgmap v448: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T20:58:18.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:18 smithi157 ceph-mon[33359]: osdmap e330: 8 total, 8 up, 8 in 2024-03-11T20:58:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi129-82567-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi129-82567-62"}]': finished 2024-03-11T20:58:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-47"}]': finished 2024-03-11T20:58:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:18 smithi157 ceph-mon[33359]: osdmap e331: 8 total, 8 up, 8 in 2024-03-11T20:58:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-47", "mode": "writeback"}]: dispatch 2024-03-11T20:58:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-47", "mode": "writeback"}]: dispatch 2024-03-11T20:58:19.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:19.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:19 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:58:19.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-47", "mode": "writeback"}]': finished 2024-03-11T20:58:19.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:58:19.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:19 smithi157 ceph-mon[33359]: osdmap e332: 8 total, 8 up, 8 in 2024-03-11T20:58:19.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:58:19.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:19.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:19 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:58:19.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-47", "mode": "writeback"}]': finished 2024-03-11T20:58:19.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:58:19.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:19 smithi129 ceph-mon[30184]: osdmap e332: 8 total, 8 up, 8 in 2024-03-11T20:58:19.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:58:19.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:19.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:19 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:58:19.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-47", "mode": "writeback"}]': finished 2024-03-11T20:58:19.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:58:19.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:19 smithi129 ceph-mon[42066]: osdmap e332: 8 total, 8 up, 8 in 2024-03-11T20:58:19.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:58:20.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:20 smithi157 ceph-mon[33359]: pgmap v451: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-11T20:58:20.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_count","val": "2"}]': finished 2024-03-11T20:58:20.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:20 smithi157 ceph-mon[33359]: osdmap e333: 8 total, 8 up, 8 in 2024-03-11T20:58:20.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:58:20.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:58:20.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi129-82567-62"}]: dispatch 2024-03-11T20:58:20.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:20 smithi129 ceph-mon[30184]: pgmap v451: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-11T20:58:20.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_count","val": "2"}]': finished 2024-03-11T20:58:20.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:20 smithi129 ceph-mon[30184]: osdmap e333: 8 total, 8 up, 8 in 2024-03-11T20:58:20.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:58:20.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:58:20.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi129-82567-62"}]: dispatch 2024-03-11T20:58:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:20 smithi129 ceph-mon[42066]: pgmap v451: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-11T20:58:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_count","val": "2"}]': finished 2024-03-11T20:58:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:20 smithi129 ceph-mon[42066]: osdmap e333: 8 total, 8 up, 8 in 2024-03-11T20:58:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:58:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:58:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi129-82567-62"}]: dispatch 2024-03-11T20:58:22.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:21 smithi129 ceph-mon[30184]: pgmap v454: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:22.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_period","val": "600"}]': finished 2024-03-11T20:58:22.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi129-82567-62"}]': finished 2024-03-11T20:58:22.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:21 smithi129 ceph-mon[30184]: osdmap e334: 8 total, 8 up, 8 in 2024-03-11T20:58:22.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi129-82567-62"}]: dispatch 2024-03-11T20:58:22.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:58:22.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:58:22.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:21 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:22.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:21 smithi129 ceph-mon[42066]: pgmap v454: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:22.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_period","val": "600"}]': finished 2024-03-11T20:58:22.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi129-82567-62"}]': finished 2024-03-11T20:58:22.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:21 smithi129 ceph-mon[42066]: osdmap e334: 8 total, 8 up, 8 in 2024-03-11T20:58:22.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi129-82567-62"}]: dispatch 2024-03-11T20:58:22.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:58:22.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:58:22.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:21 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:22.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:21 smithi157 ceph-mon[33359]: pgmap v454: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:22.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_period","val": "600"}]': finished 2024-03-11T20:58:22.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi129-82567-62"}]': finished 2024-03-11T20:58:22.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:21 smithi157 ceph-mon[33359]: osdmap e334: 8 total, 8 up, 8 in 2024-03-11T20:58:22.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi129-82567-62"}]: dispatch 2024-03-11T20:58:22.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:58:22.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:58:22.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:21 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:23.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:58:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi129-82567-62"}]': finished 2024-03-11T20:58:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T20:58:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T20:58:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: osdmap e335: 8 total, 8 up, 8 in 2024-03-11T20:58:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T20:58:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi129-82567-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi129-82567-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T20:58:23.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi129-82567-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:23.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: osdmap e336: 8 total, 8 up, 8 in 2024-03-11T20:58:23.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T20:58:23.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi129-82567-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T20:58:23.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi129-82567-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:58:23.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi129-82567-62"}]': finished 2024-03-11T20:58:23.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T20:58:23.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T20:58:23.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: osdmap e335: 8 total, 8 up, 8 in 2024-03-11T20:58:23.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T20:58:23.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi129-82567-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:23.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi129-82567-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:23.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T20:58:23.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi129-82567-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:23.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: osdmap e336: 8 total, 8 up, 8 in 2024-03-11T20:58:23.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T20:58:23.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi129-82567-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T20:58:23.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi129-82567-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:58:23.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3102296812' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi129-82567-62"}]': finished 2024-03-11T20:58:23.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T20:58:23.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T20:58:23.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: osdmap e335: 8 total, 8 up, 8 in 2024-03-11T20:58:23.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T20:58:23.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi129-82567-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:23.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi129-82567-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:23.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T20:58:23.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi129-82567-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:23.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: osdmap e336: 8 total, 8 up, 8 in 2024-03-11T20:58:23.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T20:58:23.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi129-82567-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T20:58:23.482 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi129-82567-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:23.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:58:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:58:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:24 smithi129 ceph-mon[30184]: pgmap v457: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:24 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-11T20:58:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:24 smithi129 ceph-mon[42066]: pgmap v457: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:24 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-11T20:58:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:24 smithi157 ceph-mon[33359]: pgmap v457: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:24 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-11T20:58:25.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T20:58:25.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:25 smithi129 ceph-mon[30184]: osdmap e337: 8 total, 8 up, 8 in 2024-03-11T20:58:25.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T20:58:25.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi129-82567-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi129-82567-63"}]': finished 2024-03-11T20:58:25.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-11T20:58:25.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:25 smithi129 ceph-mon[30184]: osdmap e338: 8 total, 8 up, 8 in 2024-03-11T20:58:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:25.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:25.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T20:58:25.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:25 smithi129 ceph-mon[42066]: osdmap e337: 8 total, 8 up, 8 in 2024-03-11T20:58:25.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T20:58:25.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi129-82567-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi129-82567-63"}]': finished 2024-03-11T20:58:25.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-11T20:58:25.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:25 smithi129 ceph-mon[42066]: osdmap e338: 8 total, 8 up, 8 in 2024-03-11T20:58:25.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:25.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T20:58:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:25 smithi157 ceph-mon[33359]: osdmap e337: 8 total, 8 up, 8 in 2024-03-11T20:58:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T20:58:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi129-82567-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi129-82567-63"}]': finished 2024-03-11T20:58:25.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-11T20:58:25.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:25 smithi157 ceph-mon[33359]: osdmap e338: 8 total, 8 up, 8 in 2024-03-11T20:58:25.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:25.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:26.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:58:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:58:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:58:26.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:26 smithi129 ceph-mon[30184]: pgmap v460: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:26.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:26 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:58:26.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:26 smithi129 ceph-mon[30184]: osdmap e339: 8 total, 8 up, 8 in 2024-03-11T20:58:26.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:26 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:26.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:26 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:26.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:26 smithi129 ceph-mon[42066]: pgmap v460: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:26.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:26 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:58:26.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:26 smithi129 ceph-mon[42066]: osdmap e339: 8 total, 8 up, 8 in 2024-03-11T20:58:26.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:26 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:26.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:26 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:26.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:26 smithi157 ceph-mon[33359]: pgmap v460: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:26.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:26 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:58:26.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:26 smithi157 ceph-mon[33359]: osdmap e339: 8 total, 8 up, 8 in 2024-03-11T20:58:26.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:26 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:26.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:26 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:28.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[42066]: pgmap v463: 300 pgs: 300 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T20:58:28.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47"}]': finished 2024-03-11T20:58:28.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[42066]: osdmap e340: 8 total, 8 up, 8 in 2024-03-11T20:58:28.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:28.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:28.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:28.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:28.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:28.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:28.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:28.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[30184]: pgmap v463: 300 pgs: 300 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T20:58:28.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47"}]': finished 2024-03-11T20:58:28.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[30184]: osdmap e340: 8 total, 8 up, 8 in 2024-03-11T20:58:28.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:28.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:28.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:28.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:28.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:28.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:28.481 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:27 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:27 smithi157 ceph-mon[33359]: pgmap v463: 300 pgs: 300 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T20:58:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:27 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47"}]': finished 2024-03-11T20:58:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:27 smithi157 ceph-mon[33359]: osdmap e340: 8 total, 8 up, 8 in 2024-03-11T20:58:28.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:28.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:27 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:28.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:28.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:27 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:28.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:28.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:27 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-47"}]: dispatch 2024-03-11T20:58:28.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:27 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi129-82567-63"}]': finished 2024-03-11T20:58:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:29 smithi129 ceph-mon[30184]: osdmap e341: 8 total, 8 up, 8 in 2024-03-11T20:58:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi129-82567-63"}]': finished 2024-03-11T20:58:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:29 smithi129 ceph-mon[42066]: osdmap e341: 8 total, 8 up, 8 in 2024-03-11T20:58:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi129-82567-63"}]': finished 2024-03-11T20:58:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:29 smithi157 ceph-mon[33359]: osdmap e341: 8 total, 8 up, 8 in 2024-03-11T20:58:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2718349886' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi129-82567-63"}]: dispatch 2024-03-11T20:58:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:30 smithi129 ceph-mon[30184]: pgmap v466: 260 pgs: 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T20:58:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:30 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi129-82567-63"}]': finished 2024-03-11T20:58:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:30 smithi129 ceph-mon[30184]: osdmap e342: 8 total, 8 up, 8 in 2024-03-11T20:58:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:30 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:30.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi129-82567-64"}]: dispatch 2024-03-11T20:58:30.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi129-82567-64"}]: dispatch 2024-03-11T20:58:30.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi129-82567-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:30.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:30 smithi129 ceph-mon[42066]: pgmap v466: 260 pgs: 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T20:58:30.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:30 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi129-82567-63"}]': finished 2024-03-11T20:58:30.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:30 smithi129 ceph-mon[42066]: osdmap e342: 8 total, 8 up, 8 in 2024-03-11T20:58:30.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:30.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:30 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:30.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi129-82567-64"}]: dispatch 2024-03-11T20:58:30.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi129-82567-64"}]: dispatch 2024-03-11T20:58:30.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi129-82567-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:30.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:30 smithi157 ceph-mon[33359]: pgmap v466: 260 pgs: 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T20:58:30.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:30 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi129-82567-63"}]': finished 2024-03-11T20:58:30.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:30 smithi157 ceph-mon[33359]: osdmap e342: 8 total, 8 up, 8 in 2024-03-11T20:58:30.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:30.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:30 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:30.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi129-82567-64"}]: dispatch 2024-03-11T20:58:30.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi129-82567-64"}]: dispatch 2024-03-11T20:58:30.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi129-82567-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:31.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:58:31.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi129-82567-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:31.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:31 smithi129 ceph-mon[30184]: osdmap e343: 8 total, 8 up, 8 in 2024-03-11T20:58:31.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi129-82567-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi129-82567-64"}]: dispatch 2024-03-11T20:58:31.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:31.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:31.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:58:31.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:58:31.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi129-82567-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:31.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:31 smithi129 ceph-mon[42066]: osdmap e343: 8 total, 8 up, 8 in 2024-03-11T20:58:31.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi129-82567-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi129-82567-64"}]: dispatch 2024-03-11T20:58:31.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:31.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:31.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:58:31.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:58:31.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi129-82567-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:31.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:31 smithi157 ceph-mon[33359]: osdmap e343: 8 total, 8 up, 8 in 2024-03-11T20:58:31.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi129-82567-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi129-82567-64"}]: dispatch 2024-03-11T20:58:31.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:31.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:31.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:58:32.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[30184]: pgmap v469: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:32.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:32.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[30184]: osdmap e344: 8 total, 8 up, 8 in 2024-03-11T20:58:32.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:32.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi129-82567-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi129-82567-64"}]': finished 2024-03-11T20:58:32.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-49"}]': finished 2024-03-11T20:58:32.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[30184]: osdmap e345: 8 total, 8 up, 8 in 2024-03-11T20:58:32.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-49", "mode": "readproxy"}]: dispatch 2024-03-11T20:58:32.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-49", "mode": "readproxy"}]: dispatch 2024-03-11T20:58:32.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[42066]: pgmap v469: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:32.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:32.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[42066]: osdmap e344: 8 total, 8 up, 8 in 2024-03-11T20:58:32.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:32.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi129-82567-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi129-82567-64"}]': finished 2024-03-11T20:58:32.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-49"}]': finished 2024-03-11T20:58:32.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[42066]: osdmap e345: 8 total, 8 up, 8 in 2024-03-11T20:58:32.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-49", "mode": "readproxy"}]: dispatch 2024-03-11T20:58:32.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:32 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-49", "mode": "readproxy"}]: dispatch 2024-03-11T20:58:32.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:32 smithi157 ceph-mon[33359]: pgmap v469: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:32.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:32 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:32.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:32 smithi157 ceph-mon[33359]: osdmap e344: 8 total, 8 up, 8 in 2024-03-11T20:58:32.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:32 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:32.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:32 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi129-82567-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi129-82567-64"}]': finished 2024-03-11T20:58:32.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:32 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-49"}]': finished 2024-03-11T20:58:32.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:32 smithi157 ceph-mon[33359]: osdmap e345: 8 total, 8 up, 8 in 2024-03-11T20:58:32.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:32 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-49", "mode": "readproxy"}]: dispatch 2024-03-11T20:58:32.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:32 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-49", "mode": "readproxy"}]: dispatch 2024-03-11T20:58:33.520 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:33 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:58:33.520 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-49", "mode": "readproxy"}]': finished 2024-03-11T20:58:33.521 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:33 smithi157 ceph-mon[33359]: osdmap e346: 8 total, 8 up, 8 in 2024-03-11T20:58:33.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:33 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:58:33.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-49", "mode": "readproxy"}]': finished 2024-03-11T20:58:33.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:33 smithi129 ceph-mon[30184]: osdmap e346: 8 total, 8 up, 8 in 2024-03-11T20:58:33.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:33 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:58:33.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-49", "mode": "readproxy"}]': finished 2024-03-11T20:58:33.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:33 smithi129 ceph-mon[42066]: osdmap e346: 8 total, 8 up, 8 in 2024-03-11T20:58:33.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:58:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:58:34.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:34 smithi129 ceph-mon[30184]: pgmap v472: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:58:34.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:34 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:34.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:34 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:58:34.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:34 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:34.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:34 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:34.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:34 smithi129 ceph-mon[30184]: osdmap e347: 8 total, 8 up, 8 in 2024-03-11T20:58:34.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:34 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi129-82567-64"}]: dispatch 2024-03-11T20:58:34.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:34 smithi129 ceph-mon[42066]: pgmap v472: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:58:34.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:34 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:34.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:34 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:58:34.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:34 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:34.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:34 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:34.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:34 smithi129 ceph-mon[42066]: osdmap e347: 8 total, 8 up, 8 in 2024-03-11T20:58:34.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:34 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi129-82567-64"}]: dispatch 2024-03-11T20:58:34.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:34 smithi157 ceph-mon[33359]: pgmap v472: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:58:34.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:34 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:34.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:34 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:58:34.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:34 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:34.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:34 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:34.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:34 smithi157 ceph-mon[33359]: osdmap e347: 8 total, 8 up, 8 in 2024-03-11T20:58:34.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:34 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi129-82567-64"}]: dispatch 2024-03-11T20:58:36.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:58:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:58:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:58:36.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:36 smithi129 ceph-mon[30184]: pgmap v475: 292 pgs: 292 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:58:36.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi129-82567-64"}]': finished 2024-03-11T20:58:36.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:36 smithi129 ceph-mon[30184]: osdmap e348: 8 total, 8 up, 8 in 2024-03-11T20:58:36.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi129-82567-64"}]: dispatch 2024-03-11T20:58:36.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:36 smithi129 ceph-mon[42066]: pgmap v475: 292 pgs: 292 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:58:36.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi129-82567-64"}]': finished 2024-03-11T20:58:36.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:36 smithi129 ceph-mon[42066]: osdmap e348: 8 total, 8 up, 8 in 2024-03-11T20:58:36.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi129-82567-64"}]: dispatch 2024-03-11T20:58:36.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:36 smithi157 ceph-mon[33359]: pgmap v475: 292 pgs: 292 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T20:58:36.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi129-82567-64"}]': finished 2024-03-11T20:58:36.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:36 smithi157 ceph-mon[33359]: osdmap e348: 8 total, 8 up, 8 in 2024-03-11T20:58:36.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi129-82567-64"}]: dispatch 2024-03-11T20:58:37.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi129-82567-64"}]': finished 2024-03-11T20:58:37.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:37 smithi129 ceph-mon[30184]: osdmap e349: 8 total, 8 up, 8 in 2024-03-11T20:58:37.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:37.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:37.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:37.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:37.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi129-82567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:37.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi129-82567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:37.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi129-82567-64"}]': finished 2024-03-11T20:58:37.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:37 smithi129 ceph-mon[42066]: osdmap e349: 8 total, 8 up, 8 in 2024-03-11T20:58:37.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:37.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:37.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:37.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:37.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi129-82567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:37.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi129-82567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:37.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/275282825' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi129-82567-64"}]': finished 2024-03-11T20:58:37.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:37 smithi157 ceph-mon[33359]: osdmap e349: 8 total, 8 up, 8 in 2024-03-11T20:58:37.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:37.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:37.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:37.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:37.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi129-82567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:37.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi129-82567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:38.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:38 smithi129 ceph-mon[30184]: pgmap v478: 292 pgs: 292 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T20:58:38.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi129-82567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:38.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:38 smithi129 ceph-mon[30184]: osdmap e350: 8 total, 8 up, 8 in 2024-03-11T20:58:38.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi129-82567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:38.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi129-82567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:38.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:38 smithi129 ceph-mon[42066]: pgmap v478: 292 pgs: 292 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T20:58:38.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi129-82567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:38.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:38 smithi129 ceph-mon[42066]: osdmap e350: 8 total, 8 up, 8 in 2024-03-11T20:58:38.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi129-82567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:38.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi129-82567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:38.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:38 smithi157 ceph-mon[33359]: pgmap v478: 292 pgs: 292 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T20:58:38.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi129-82567-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:38.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:38 smithi157 ceph-mon[33359]: osdmap e350: 8 total, 8 up, 8 in 2024-03-11T20:58:38.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi129-82567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:38.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi129-82567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:39.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:39 smithi129 ceph-mon[30184]: osdmap e351: 8 total, 8 up, 8 in 2024-03-11T20:58:39.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:39 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi129-82567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi129-82567-65"}]': finished 2024-03-11T20:58:39.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:39 smithi129 ceph-mon[30184]: osdmap e352: 8 total, 8 up, 8 in 2024-03-11T20:58:39.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:39 smithi129 ceph-mon[42066]: osdmap e351: 8 total, 8 up, 8 in 2024-03-11T20:58:39.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:39 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi129-82567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi129-82567-65"}]': finished 2024-03-11T20:58:39.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:39 smithi129 ceph-mon[42066]: osdmap e352: 8 total, 8 up, 8 in 2024-03-11T20:58:39.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:39 smithi157 ceph-mon[33359]: osdmap e351: 8 total, 8 up, 8 in 2024-03-11T20:58:39.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:39 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi129-82567-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi129-82567-65"}]': finished 2024-03-11T20:58:39.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:39 smithi157 ceph-mon[33359]: osdmap e352: 8 total, 8 up, 8 in 2024-03-11T20:58:40.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:40 smithi129 ceph-mon[30184]: pgmap v481: 292 pgs: 292 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T20:58:40.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:40 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:40.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:40 smithi129 ceph-mon[30184]: osdmap e353: 8 total, 8 up, 8 in 2024-03-11T20:58:40.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:40 smithi129 ceph-mon[42066]: pgmap v481: 292 pgs: 292 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T20:58:40.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:40 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:40.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:40 smithi129 ceph-mon[42066]: osdmap e353: 8 total, 8 up, 8 in 2024-03-11T20:58:40.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:40 smithi157 ceph-mon[33359]: pgmap v481: 292 pgs: 292 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T20:58:40.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:40 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:40.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:40 smithi157 ceph-mon[33359]: osdmap e353: 8 total, 8 up, 8 in 2024-03-11T20:58:42.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:42 smithi129 ceph-mon[30184]: pgmap v484: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:42.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:42 smithi129 ceph-mon[30184]: osdmap e354: 8 total, 8 up, 8 in 2024-03-11T20:58:42.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:42.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:42 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:42.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:42 smithi129 ceph-mon[42066]: pgmap v484: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:42.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:42 smithi129 ceph-mon[42066]: osdmap e354: 8 total, 8 up, 8 in 2024-03-11T20:58:42.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:42.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:42 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:42.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:42 smithi157 ceph-mon[33359]: pgmap v484: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:42.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:42 smithi157 ceph-mon[33359]: osdmap e354: 8 total, 8 up, 8 in 2024-03-11T20:58:42.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:42.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:42 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:43.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi129-82567-65"}]': finished 2024-03-11T20:58:43.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:43 smithi129 ceph-mon[30184]: osdmap e355: 8 total, 8 up, 8 in 2024-03-11T20:58:43.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:43.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:43.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi129-82567-65"}]': finished 2024-03-11T20:58:43.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:43 smithi129 ceph-mon[30184]: osdmap e356: 8 total, 8 up, 8 in 2024-03-11T20:58:43.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi129-82567-65"}]': finished 2024-03-11T20:58:43.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:43 smithi129 ceph-mon[42066]: osdmap e355: 8 total, 8 up, 8 in 2024-03-11T20:58:43.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:43.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:43.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi129-82567-65"}]': finished 2024-03-11T20:58:43.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:43 smithi129 ceph-mon[42066]: osdmap e356: 8 total, 8 up, 8 in 2024-03-11T20:58:43.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi129-82567-65"}]': finished 2024-03-11T20:58:43.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:43 smithi157 ceph-mon[33359]: osdmap e355: 8 total, 8 up, 8 in 2024-03-11T20:58:43.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2635165755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:43.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi129-82567-65"}]: dispatch 2024-03-11T20:58:43.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi129-82567-65"}]': finished 2024-03-11T20:58:43.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:43 smithi157 ceph-mon[33359]: osdmap e356: 8 total, 8 up, 8 in 2024-03-11T20:58:43.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:58:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:58:44.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:44 smithi129 ceph-mon[30184]: pgmap v487: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:44.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi129-82567-66"}]: dispatch 2024-03-11T20:58:44.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi129-82567-66"}]: dispatch 2024-03-11T20:58:44.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi129-82567-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:44.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:44.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:44 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:44.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:44 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:44.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:44 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:58:44.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:44 smithi129 ceph-mon[42066]: pgmap v487: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:44.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi129-82567-66"}]: dispatch 2024-03-11T20:58:44.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi129-82567-66"}]: dispatch 2024-03-11T20:58:44.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi129-82567-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:44.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:44.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:44 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:44.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:44 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:44.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:44 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:58:44.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:44 smithi157 ceph-mon[33359]: pgmap v487: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:44.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi129-82567-66"}]: dispatch 2024-03-11T20:58:44.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi129-82567-66"}]: dispatch 2024-03-11T20:58:44.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi129-82567-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T20:58:44.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:44.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:44 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:44.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:44 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:44.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:44 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:58:45.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi129-82567-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:45.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:58:45.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[30184]: osdmap e357: 8 total, 8 up, 8 in 2024-03-11T20:58:45.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi129-82567-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi129-82567-66"}]: dispatch 2024-03-11T20:58:45.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:45.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:45.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:58:45.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:58:45.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:58:45.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:58:45.731 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49"}]': finished 2024-03-11T20:58:45.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi129-82567-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:45.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:58:45.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[42066]: osdmap e357: 8 total, 8 up, 8 in 2024-03-11T20:58:45.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi129-82567-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi129-82567-66"}]: dispatch 2024-03-11T20:58:45.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:45.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:45.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:58:45.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:58:45.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:58:45.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:58:45.732 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:45 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49"}]': finished 2024-03-11T20:58:45.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi129-82567-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T20:58:45.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:45 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:58:45.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:45 smithi157 ceph-mon[33359]: osdmap e357: 8 total, 8 up, 8 in 2024-03-11T20:58:45.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi129-82567-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi129-82567-66"}]: dispatch 2024-03-11T20:58:45.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:45.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:45 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:45.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:45 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:58:45.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:45 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:58:45.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:45 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:58:45.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:45 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:58:45.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:45 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49"}]': finished 2024-03-11T20:58:46.063 INFO:tasks.workunit.client.0.smithi129.stdout:63:head 2024-03-11T20:58:46.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:5d165639:::164:head 2024-03-11T20:58:46.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:f43765fc:::165:head 2024-03-11T20:58:46.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:b4c720e9:::166:head 2024-03-11T20:58:46.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:e694b040:::167:head 2024-03-11T20:58:46.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:afa38db2:::168:head 2024-03-11T20:58:46.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:77ba9f53:::169:head 2024-03-11T20:58:46.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:87495034:::170:head 2024-03-11T20:58:46.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:7c96bf0e:::171:head 2024-03-11T20:58:46.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:dbe346cc:::172:head 2024-03-11T20:58:46.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:e943ec24:::173:head 2024-03-11T20:58:46.064 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:f97a9c0c:::174:head 2024-03-11T20:58:46.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:6f26e74d:::175:head 2024-03-11T20:58:46.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:4f95e106:::176:head 2024-03-11T20:58:46.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:0e6f2f8f:::177:head 2024-03-11T20:58:46.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:05db05f1:::178:head 2024-03-11T20:58:46.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:38a78d66:::179:head 2024-03-11T20:58:46.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:d095610b:::180:head 2024-03-11T20:58:46.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:a1a9d709:::181:head 2024-03-11T20:58:46.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:1e5d39db:::182:head 2024-03-11T20:58:46.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:f7df4fb9:::183:head 2024-03-11T20:58:46.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:03a7f161:::184:head 2024-03-11T20:58:46.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:ba70721e:::185:head 2024-03-11T20:58:46.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:28e5662d:::186:head 2024-03-11T20:58:46.065 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:973d52de:::187:head 2024-03-11T20:58:46.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:4303eb1c:::188:head 2024-03-11T20:58:46.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:b990b48e:::189:head 2024-03-11T20:58:46.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:29b8165b:::190:head 2024-03-11T20:58:46.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:3547f197:::191:head 2024-03-11T20:58:46.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:7e260936:::192:head 2024-03-11T20:58:46.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:1abec7b1:::193:head 2024-03-11T20:58:46.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:10fdda93:::194:head 2024-03-11T20:58:46.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:15817eea:::195:head 2024-03-11T20:58:46.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:770bab57:::196:head 2024-03-11T20:58:46.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:ed9e13e7:::197:head 2024-03-11T20:58:46.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:71471a8f:::198:head 2024-03-11T20:58:46.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: checking for 276:10fb1d02:::199:head 2024-03-11T20:58:46.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8298 ms) 2024-03-11T20:58:46.066 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-03-11T20:58:46.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710190680,0 2024-03-11T20:58:46.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: first is 1710190680 2024-03-11T20:58:46.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710190680,0 2024-03-11T20:58:46.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710190680,0 2024-03-11T20:58:46.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710190680,1710190682,1710190683,0 2024-03-11T20:58:46.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710190680,1710190682,1710190683,0 2024-03-11T20:58:46.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710190680,1710190682,1710190683,0 2024-03-11T20:58:46.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710190680,1710190682,1710190683,1710190685,1710190686,0 2024-03-11T20:58:46.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710190680,1710190682,1710190683,1710190685,1710190686,0 2024-03-11T20:58:46.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710190680,1710190682,1710190683,1710190685,1710190686,0 2024-03-11T20:58:46.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710190680,1710190682,1710190683,1710190685,1710190686,1710190688,1710190689,0 2024-03-11T20:58:46.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710190680,1710190682,1710190683,1710190685,1710190686,1710190688,1710190689,0 2024-03-11T20:58:46.067 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710190680,1710190682,1710190683,1710190685,1710190686,1710190688,1710190689,0 2024-03-11T20:58:46.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710190683,1710190685,1710190686,1710190688,1710190689,1710190691,1710190692,0 2024-03-11T20:58:46.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: first now 1710190683, trimmed 2024-03-11T20:58:46.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20862 ms) 2024-03-11T20:58:46.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-03-11T20:58:46.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: foo0 2024-03-11T20:58:46.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-11T20:58:46.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14030 ms) 2024-03-11T20:58:46.068 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-03-11T20:58:46.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:58:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:58:46.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-mon[30184]: pgmap v490: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:46.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:46.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-mon[30184]: osdmap e358: 8 total, 8 up, 8 in 2024-03-11T20:58:46.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:46.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:46.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:46.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi129-82567-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi129-82567-66"}]': finished 2024-03-11T20:58:46.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-mon[30184]: osdmap e359: 8 total, 8 up, 8 in 2024-03-11T20:58:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-mon[42066]: pgmap v490: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-mon[42066]: osdmap e358: 8 total, 8 up, 8 in 2024-03-11T20:58:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi129-82567-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi129-82567-66"}]': finished 2024-03-11T20:58:46.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:46 smithi129 ceph-mon[42066]: osdmap e359: 8 total, 8 up, 8 in 2024-03-11T20:58:46.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:46 smithi157 ceph-mon[33359]: pgmap v490: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:46.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:46 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:46.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:46 smithi157 ceph-mon[33359]: osdmap e358: 8 total, 8 up, 8 in 2024-03-11T20:58:46.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:46 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:58:46.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:46 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:46.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:46 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-49"}]: dispatch 2024-03-11T20:58:46.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:46 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi129-82567-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi129-82567-66"}]': finished 2024-03-11T20:58:46.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:46 smithi157 ceph-mon[33359]: osdmap e359: 8 total, 8 up, 8 in 2024-03-11T20:58:48.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:48 smithi129 ceph-mon[30184]: pgmap v493: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:48.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:48 smithi129 ceph-mon[30184]: osdmap e360: 8 total, 8 up, 8 in 2024-03-11T20:58:48.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:48.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:48 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:48.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:48 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:48.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:48 smithi129 ceph-mon[42066]: pgmap v493: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:48.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:48 smithi129 ceph-mon[42066]: osdmap e360: 8 total, 8 up, 8 in 2024-03-11T20:58:48.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:48.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:48 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:48.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:48 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:48.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:48 smithi157 ceph-mon[33359]: pgmap v493: 268 pgs: 8 creating+peering, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:48.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:48 smithi157 ceph-mon[33359]: osdmap e360: 8 total, 8 up, 8 in 2024-03-11T20:58:48.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:48.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:48 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:58:48.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:48 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:49.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:58:49.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[30184]: osdmap e361: 8 total, 8 up, 8 in 2024-03-11T20:58:49.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi129-82567-66"}]: dispatch 2024-03-11T20:58:49.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:49.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:49.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:58:49.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:49.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi129-82567-66"}]': finished 2024-03-11T20:58:49.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:58:49.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[30184]: osdmap e362: 8 total, 8 up, 8 in 2024-03-11T20:58:49.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi129-82567-66"}]: dispatch 2024-03-11T20:58:49.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:58:49.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[42066]: osdmap e361: 8 total, 8 up, 8 in 2024-03-11T20:58:49.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi129-82567-66"}]: dispatch 2024-03-11T20:58:49.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:49.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:49.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:58:49.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:49.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi129-82567-66"}]': finished 2024-03-11T20:58:49.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:58:49.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[42066]: osdmap e362: 8 total, 8 up, 8 in 2024-03-11T20:58:49.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi129-82567-66"}]: dispatch 2024-03-11T20:58:49.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:58:49.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:49 smithi157 ceph-mon[33359]: osdmap e361: 8 total, 8 up, 8 in 2024-03-11T20:58:49.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi129-82567-66"}]: dispatch 2024-03-11T20:58:49.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:49.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T20:58:49.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:49 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:58:49.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:49 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:58:49.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi129-82567-66"}]': finished 2024-03-11T20:58:49.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T20:58:49.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:49 smithi157 ceph-mon[33359]: osdmap e362: 8 total, 8 up, 8 in 2024-03-11T20:58:49.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi129-82567-66"}]: dispatch 2024-03-11T20:58:50.075 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3032 ms) 2024-03-11T20:58:50.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-03-11T20:58:50.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7025 ms) 2024-03-11T20:58:50.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-03-11T20:58:50.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7017 ms) 2024-03-11T20:58:50.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-03-11T20:58:50.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7034 ms) 2024-03-11T20:58:50.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-03-11T20:58:50.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7013 ms) 2024-03-11T20:58:50.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-03-11T20:58:50.076 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7016 ms) 2024-03-11T20:58:50.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-03-11T20:58:50.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7026 ms) 2024-03-11T20:58:50.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139100 ms total) 2024-03-11T20:58:50.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: 2024-03-11T20:58:50.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-03-11T20:58:50.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (296409 ms total) 2024-03-11T20:58:50.077 INFO:tasks.workunit.client.0.smithi129.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-03-11T20:58:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:50 smithi129 ceph-mon[30184]: pgmap v496: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:58:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:58:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi129-82567-66"}]': finished 2024-03-11T20:58:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-51"}]': finished 2024-03-11T20:58:50.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:50 smithi129 ceph-mon[30184]: osdmap e363: 8 total, 8 up, 8 in 2024-03-11T20:58:50.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:50 smithi129 ceph-mon[42066]: pgmap v496: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:50.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:58:50.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:58:50.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi129-82567-66"}]': finished 2024-03-11T20:58:50.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-51"}]': finished 2024-03-11T20:58:50.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:50 smithi129 ceph-mon[42066]: osdmap e363: 8 total, 8 up, 8 in 2024-03-11T20:58:50.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:50 smithi157 ceph-mon[33359]: pgmap v496: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:50.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:58:50.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:58:50.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3732857467' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi129-82567-66"}]': finished 2024-03-11T20:58:50.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-51"}]': finished 2024-03-11T20:58:50.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:50 smithi157 ceph-mon[33359]: osdmap e363: 8 total, 8 up, 8 in 2024-03-11T20:58:51.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-51", "mode": "writeback"}]: dispatch 2024-03-11T20:58:51.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:51 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-51", "mode": "writeback"}]: dispatch 2024-03-11T20:58:51.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-51", "mode": "writeback"}]: dispatch 2024-03-11T20:58:51.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:51 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-51", "mode": "writeback"}]: dispatch 2024-03-11T20:58:51.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-51", "mode": "writeback"}]: dispatch 2024-03-11T20:58:51.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:51 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-51", "mode": "writeback"}]: dispatch 2024-03-11T20:58:52.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:52 smithi129 ceph-mon[30184]: pgmap v499: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:52.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:52 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:58:52.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-51", "mode": "writeback"}]': finished 2024-03-11T20:58:52.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:52 smithi129 ceph-mon[30184]: osdmap e364: 8 total, 8 up, 8 in 2024-03-11T20:58:52.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:58:52.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:58:52.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:52 smithi129 ceph-mon[42066]: pgmap v499: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:52.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:52 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:58:52.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-51", "mode": "writeback"}]': finished 2024-03-11T20:58:52.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:52 smithi129 ceph-mon[42066]: osdmap e364: 8 total, 8 up, 8 in 2024-03-11T20:58:52.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:58:52.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:58:52.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:52 smithi157 ceph-mon[33359]: pgmap v499: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:58:52.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:52 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T20:58:52.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-51", "mode": "writeback"}]': finished 2024-03-11T20:58:52.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:52 smithi157 ceph-mon[33359]: osdmap e364: 8 total, 8 up, 8 in 2024-03-11T20:58:52.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:58:52.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T20:58:53.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:53 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_count","val": "2"}]': finished 2024-03-11T20:58:53.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:53 smithi129 ceph-mon[30184]: osdmap e365: 8 total, 8 up, 8 in 2024-03-11T20:58:53.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:53 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:58:53.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:53 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:58:53.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:53 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_count","val": "2"}]': finished 2024-03-11T20:58:53.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:53 smithi129 ceph-mon[42066]: osdmap e365: 8 total, 8 up, 8 in 2024-03-11T20:58:53.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:53 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:58:53.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:53 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:58:53.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:53 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_count","val": "2"}]': finished 2024-03-11T20:58:53.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:53 smithi157 ceph-mon[33359]: osdmap e365: 8 total, 8 up, 8 in 2024-03-11T20:58:53.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:53 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:58:53.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:53 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T20:58:53.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:58:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:58:54.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:54 smithi129 ceph-mon[30184]: pgmap v502: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-11T20:58:54.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_period","val": "600"}]': finished 2024-03-11T20:58:54.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:54 smithi129 ceph-mon[30184]: osdmap e366: 8 total, 8 up, 8 in 2024-03-11T20:58:54.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:58:54.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:58:54.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:54 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:54.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:54 smithi129 ceph-mon[42066]: pgmap v502: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-11T20:58:54.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_period","val": "600"}]': finished 2024-03-11T20:58:54.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:54 smithi129 ceph-mon[42066]: osdmap e366: 8 total, 8 up, 8 in 2024-03-11T20:58:54.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:58:54.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:58:54.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:54 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:54.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:54 smithi157 ceph-mon[33359]: pgmap v502: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-11T20:58:54.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_period","val": "600"}]': finished 2024-03-11T20:58:54.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:54 smithi157 ceph-mon[33359]: osdmap e366: 8 total, 8 up, 8 in 2024-03-11T20:58:54.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:58:54.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T20:58:54.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:54 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:58:55.380 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:55 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:58:55.381 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T20:58:55.381 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:55 smithi129 ceph-mon[30184]: osdmap e367: 8 total, 8 up, 8 in 2024-03-11T20:58:55.381 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T20:58:55.381 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T20:58:55.381 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:55 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:55.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:55 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:58:55.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T20:58:55.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:55 smithi129 ceph-mon[42066]: osdmap e367: 8 total, 8 up, 8 in 2024-03-11T20:58:55.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T20:58:55.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T20:58:55.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:55 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:55.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:55 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T20:58:55.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T20:58:55.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:55 smithi157 ceph-mon[33359]: osdmap e367: 8 total, 8 up, 8 in 2024-03-11T20:58:55.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T20:58:55.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T20:58:55.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:55 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:58:56.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:58:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:58:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:58:56.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:56 smithi129 ceph-mon[30184]: pgmap v505: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-11T20:58:56.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:56 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T20:58:56.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:56 smithi129 ceph-mon[30184]: osdmap e368: 8 total, 8 up, 8 in 2024-03-11T20:58:56.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T20:58:56.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:56 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T20:58:56.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:56 smithi129 ceph-mon[42066]: pgmap v505: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-11T20:58:56.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:56 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T20:58:56.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:56 smithi129 ceph-mon[42066]: osdmap e368: 8 total, 8 up, 8 in 2024-03-11T20:58:56.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T20:58:56.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:56 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T20:58:56.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:56 smithi157 ceph-mon[33359]: pgmap v505: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-11T20:58:56.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:56 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T20:58:56.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:56 smithi157 ceph-mon[33359]: osdmap e368: 8 total, 8 up, 8 in 2024-03-11T20:58:56.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T20:58:56.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:56 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T20:58:57.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "target_max_objects","val": "1"}]': finished 2024-03-11T20:58:57.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:57 smithi129 ceph-mon[30184]: osdmap e369: 8 total, 8 up, 8 in 2024-03-11T20:58:57.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "target_max_objects","val": "1"}]': finished 2024-03-11T20:58:57.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:57 smithi129 ceph-mon[42066]: osdmap e369: 8 total, 8 up, 8 in 2024-03-11T20:58:57.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-51","var": "target_max_objects","val": "1"}]': finished 2024-03-11T20:58:57.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:57 smithi157 ceph-mon[33359]: osdmap e369: 8 total, 8 up, 8 in 2024-03-11T20:58:58.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:58 smithi129 ceph-mon[30184]: pgmap v508: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:58.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:58:58 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-11T20:58:58.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:58 smithi129 ceph-mon[42066]: pgmap v508: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:58.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:58:58 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-11T20:58:58.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:58 smithi157 ceph-mon[33359]: pgmap v508: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T20:58:58.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:58:58 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-11T20:59:00.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:00 smithi129 ceph-mon[30184]: pgmap v509: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-03-11T20:59:00.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:00 smithi129 ceph-mon[42066]: pgmap v509: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-03-11T20:59:00.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:00 smithi157 ceph-mon[33359]: pgmap v509: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-03-11T20:59:02.137 INFO:tasks.workunit.client.0.smithi129.stdout: started 0x558254a3fba0 2024-03-11T20:59:02.138 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3fcf0 2024-03-11T20:59:02.138 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3fe00 2024-03-11T20:59:02.138 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3ff10 2024-03-11T20:59:02.138 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a40650 2024-03-11T20:59:02.138 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a401a0 2024-03-11T20:59:02.138 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a402f0 2024-03-11T20:59:02.138 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a40420 2024-03-11T20:59:02.138 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a407c0 2024-03-11T20:59:02.138 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a40930 2024-03-11T20:59:02.138 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a40f30 2024-03-11T20:59:02.138 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a40a80 2024-03-11T20:59:02.138 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a40bb0 2024-03-11T20:59:02.139 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a40cc0 2024-03-11T20:59:02.139 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a40df0 2024-03-11T20:59:02.139 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a41530 2024-03-11T20:59:02.139 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a41080 2024-03-11T20:59:02.139 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a40530 2024-03-11T20:59:02.139 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a41190 2024-03-11T20:59:02.139 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a412c0 2024-03-11T20:59:02.139 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a41b10 2024-03-11T20:59:02.139 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a41660 2024-03-11T20:59:02.139 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a41790 2024-03-11T20:59:02.139 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a418e0 2024-03-11T20:59:02.139 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a41c60 2024-03-11T20:59:02.139 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a42130 2024-03-11T20:59:02.140 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a42260 2024-03-11T20:59:02.140 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3e760 2024-03-11T20:59:02.140 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3e890 2024-03-11T20:59:02.140 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a3e9a0 2024-03-11T20:59:02.140 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a42370 2024-03-11T20:59:02.140 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a42860 2024-03-11T20:59:02.140 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a42990 2024-03-11T20:59:02.140 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a42f50 2024-03-11T20:59:02.140 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a424c0 2024-03-11T20:59:02.140 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a425d0 2024-03-11T20:59:02.140 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a42720 2024-03-11T20:59:02.140 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a430a0 2024-03-11T20:59:02.140 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a42aa0 2024-03-11T20:59:02.140 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a42bd0 2024-03-11T20:59:02.140 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a42d40 2024-03-11T20:59:02.141 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a431b0 2024-03-11T20:59:02.141 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a43680 2024-03-11T20:59:02.141 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a437b0 2024-03-11T20:59:02.141 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a41d90 2024-03-11T20:59:02.141 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a41ec0 2024-03-11T20:59:02.141 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a41ff0 2024-03-11T20:59:02.141 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a438e0 2024-03-11T20:59:02.141 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a43d90 2024-03-11T20:59:02.141 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a43ec0 2024-03-11T20:59:02.141 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a432e0 2024-03-11T20:59:02.141 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a43430 2024-03-11T20:59:02.141 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a43540 2024-03-11T20:59:02.141 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a43ff0 2024-03-11T20:59:02.141 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a444c0 2024-03-11T20:59:02.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a44610 2024-03-11T20:59:02.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a44100 2024-03-11T20:59:02.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a44250 2024-03-11T20:59:02.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a443a0 2024-03-11T20:59:02.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a44720 2024-03-11T20:59:02.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a44890 2024-03-11T20:59:02.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a44e90 2024-03-11T20:59:02.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a449e0 2024-03-11T20:59:02.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a44b30 2024-03-11T20:59:02.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a44c80 2024-03-11T20:59:02.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a44fc0 2024-03-11T20:59:02.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a454d0 2024-03-11T20:59:02.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a45620 2024-03-11T20:59:02.142 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a439f0 2024-03-11T20:59:02.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a43b20 2024-03-11T20:59:02.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a43c70 2024-03-11T20:59:02.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a45730 2024-03-11T20:59:02.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a45880 2024-03-11T20:59:02.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a45e80 2024-03-11T20:59:02.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a459d0 2024-03-11T20:59:02.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a45b00 2024-03-11T20:59:02.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a45c50 2024-03-11T20:59:02.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a45fb0 2024-03-11T20:59:02.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a464d0 2024-03-11T20:59:02.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a46600 2024-03-11T20:59:02.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a45d60 2024-03-11T20:59:02.143 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a46730 2024-03-11T20:59:02.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:02 smithi129 ceph-mon[30184]: pgmap v510: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 758 B/s rd, 0 op/s 2024-03-11T20:59:02.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:02 smithi129 ceph-mon[42066]: pgmap v510: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 758 B/s rd, 0 op/s 2024-03-11T20:59:02.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:02 smithi157 ceph-mon[33359]: pgmap v510: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 758 B/s rd, 0 op/s 2024-03-11T20:59:03.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:59:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:59:04.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:04 smithi129 ceph-mon[30184]: pgmap v511: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:04.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:59:04.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:04.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:04 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:04.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:04 smithi129 ceph-mon[42066]: pgmap v511: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:59:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:04 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:04.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:04 smithi157 ceph-mon[33359]: pgmap v511: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:04.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:59:04.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:04.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:04 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:06.463 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:59:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:59:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:59:06.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:06 smithi129 ceph-mon[30184]: pgmap v512: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:06.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:06.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:06 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:06.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:06 smithi129 ceph-mon[42066]: pgmap v512: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:06.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:06.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:06 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:06.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:06 smithi157 ceph-mon[33359]: pgmap v512: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:06.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:06.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:06 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:07.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:59:07.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:07 smithi129 ceph-mon[30184]: osdmap e370: 8 total, 8 up, 8 in 2024-03-11T20:59:07.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:59:07.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:59:07.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:59:07.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:07 smithi129 ceph-mon[42066]: osdmap e370: 8 total, 8 up, 8 in 2024-03-11T20:59:07.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:59:07.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:59:07.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T20:59:07.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:07 smithi157 ceph-mon[33359]: osdmap e370: 8 total, 8 up, 8 in 2024-03-11T20:59:07.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:59:07.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:59:08.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:08 smithi129 ceph-mon[30184]: pgmap v514: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:08.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51"}]': finished 2024-03-11T20:59:08.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:08 smithi129 ceph-mon[30184]: osdmap e371: 8 total, 8 up, 8 in 2024-03-11T20:59:08.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:08.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:08.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:59:08.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:59:08.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:08 smithi129 ceph-mon[42066]: pgmap v514: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:08.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51"}]': finished 2024-03-11T20:59:08.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:08 smithi129 ceph-mon[42066]: osdmap e371: 8 total, 8 up, 8 in 2024-03-11T20:59:08.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:08.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:08.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:59:08.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:59:08.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:08 smithi157 ceph-mon[33359]: pgmap v514: 292 pgs: 292 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:08.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51"}]': finished 2024-03-11T20:59:08.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:08 smithi157 ceph-mon[33359]: osdmap e371: 8 total, 8 up, 8 in 2024-03-11T20:59:08.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:08.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:08.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:59:08.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-51"}]: dispatch 2024-03-11T20:59:09.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:09 smithi129 ceph-mon[30184]: osdmap e372: 8 total, 8 up, 8 in 2024-03-11T20:59:09.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:09 smithi129 ceph-mon[42066]: osdmap e372: 8 total, 8 up, 8 in 2024-03-11T20:59:09.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:09 smithi157 ceph-mon[33359]: osdmap e372: 8 total, 8 up, 8 in 2024-03-11T20:59:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:10 smithi129 ceph-mon[30184]: pgmap v517: 260 pgs: 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T20:59:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:10 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-11T20:59:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:10 smithi129 ceph-mon[30184]: osdmap e373: 8 total, 8 up, 8 in 2024-03-11T20:59:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:10.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:10.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:10 smithi129 ceph-mon[42066]: pgmap v517: 260 pgs: 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T20:59:10.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:10 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-11T20:59:10.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:10 smithi129 ceph-mon[42066]: osdmap e373: 8 total, 8 up, 8 in 2024-03-11T20:59:10.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:10.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:10.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:10 smithi157 ceph-mon[33359]: pgmap v517: 260 pgs: 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-11T20:59:10.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:10 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-11T20:59:10.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:10 smithi157 ceph-mon[33359]: osdmap e373: 8 total, 8 up, 8 in 2024-03-11T20:59:10.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:10.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:11.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:11.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:11 smithi129 ceph-mon[30184]: osdmap e374: 8 total, 8 up, 8 in 2024-03-11T20:59:11.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:11.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:11.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-53"}]: dispatch 2024-03-11T20:59:11.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-53"}]: dispatch 2024-03-11T20:59:11.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:11.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:11 smithi129 ceph-mon[42066]: osdmap e374: 8 total, 8 up, 8 in 2024-03-11T20:59:11.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:11.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:11.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-53"}]: dispatch 2024-03-11T20:59:11.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-53"}]: dispatch 2024-03-11T20:59:11.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:11.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:11 smithi157 ceph-mon[33359]: osdmap e374: 8 total, 8 up, 8 in 2024-03-11T20:59:11.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:11.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:11.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-53"}]: dispatch 2024-03-11T20:59:11.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-53"}]: dispatch 2024-03-11T20:59:12.888 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:12 smithi157 ceph-mon[33359]: pgmap v520: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:59:12.888 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:12 smithi157 ceph-mon[33359]: osdmap e375: 8 total, 8 up, 8 in 2024-03-11T20:59:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:12 smithi129 ceph-mon[30184]: pgmap v520: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:59:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:12 smithi129 ceph-mon[30184]: osdmap e375: 8 total, 8 up, 8 in 2024-03-11T20:59:12.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:12 smithi129 ceph-mon[42066]: pgmap v520: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:59:12.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:12 smithi129 ceph-mon[42066]: osdmap e375: 8 total, 8 up, 8 in 2024-03-11T20:59:13.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:13 smithi129 ceph-mon[30184]: osdmap e376: 8 total, 8 up, 8 in 2024-03-11T20:59:13.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:13.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:13 smithi129 ceph-mon[42066]: osdmap e376: 8 total, 8 up, 8 in 2024-03-11T20:59:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:13.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:59:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:59:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:13 smithi157 ceph-mon[33359]: osdmap e376: 8 total, 8 up, 8 in 2024-03-11T20:59:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:14.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[30184]: pgmap v523: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:59:14.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:59:14.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:14.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[30184]: osdmap e377: 8 total, 8 up, 8 in 2024-03-11T20:59:14.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T20:59:14.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:14.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:14.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:14.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-55"}]: dispatch 2024-03-11T20:59:14.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-55"}]: dispatch 2024-03-11T20:59:14.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[42066]: pgmap v523: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:59:14.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:59:14.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:14.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[42066]: osdmap e377: 8 total, 8 up, 8 in 2024-03-11T20:59:14.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T20:59:14.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:14.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:14.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:14.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-55"}]: dispatch 2024-03-11T20:59:14.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-55"}]: dispatch 2024-03-11T20:59:14.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:14 smithi157 ceph-mon[33359]: pgmap v523: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:59:14.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:14 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:59:14.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:14.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:14 smithi157 ceph-mon[33359]: osdmap e377: 8 total, 8 up, 8 in 2024-03-11T20:59:14.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T20:59:14.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:14 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:14.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:14.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:14.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-55"}]: dispatch 2024-03-11T20:59:14.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-55"}]: dispatch 2024-03-11T20:59:15.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:15 smithi129 ceph-mon[30184]: osdmap e378: 8 total, 8 up, 8 in 2024-03-11T20:59:15.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:15 smithi129 ceph-mon[42066]: osdmap e378: 8 total, 8 up, 8 in 2024-03-11T20:59:15.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:15 smithi157 ceph-mon[33359]: osdmap e378: 8 total, 8 up, 8 in 2024-03-11T20:59:16.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:59:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:59:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:59:16.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:16 smithi129 ceph-mon[30184]: pgmap v526: 260 pgs: 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:59:16.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:16 smithi129 ceph-mon[30184]: osdmap e379: 8 total, 8 up, 8 in 2024-03-11T20:59:16.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:16.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:16 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:16.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:16 smithi129 ceph-mon[42066]: pgmap v526: 260 pgs: 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:59:16.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:16 smithi129 ceph-mon[42066]: osdmap e379: 8 total, 8 up, 8 in 2024-03-11T20:59:16.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:16.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:16 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:16.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:16 smithi157 ceph-mon[33359]: pgmap v526: 260 pgs: 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:59:16.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:16 smithi157 ceph-mon[33359]: osdmap e379: 8 total, 8 up, 8 in 2024-03-11T20:59:16.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:16.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:16 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:17.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:17.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:17 smithi129 ceph-mon[30184]: osdmap e380: 8 total, 8 up, 8 in 2024-03-11T20:59:17.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:17.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:17.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-57"}]: dispatch 2024-03-11T20:59:17.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-57"}]: dispatch 2024-03-11T20:59:17.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:17.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:17 smithi129 ceph-mon[42066]: osdmap e380: 8 total, 8 up, 8 in 2024-03-11T20:59:17.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:17.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:17.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-57"}]: dispatch 2024-03-11T20:59:17.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-57"}]: dispatch 2024-03-11T20:59:17.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:17.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:17 smithi157 ceph-mon[33359]: osdmap e380: 8 total, 8 up, 8 in 2024-03-11T20:59:17.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:17.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:17.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-57"}]: dispatch 2024-03-11T20:59:17.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-57"}]: dispatch 2024-03-11T20:59:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:18 smithi129 ceph-mon[30184]: pgmap v529: 292 pgs: 292 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-11T20:59:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:18 smithi129 ceph-mon[30184]: osdmap e381: 8 total, 8 up, 8 in 2024-03-11T20:59:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:18 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:59:18.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:18 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:18.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:18 smithi129 ceph-mon[42066]: pgmap v529: 292 pgs: 292 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-11T20:59:18.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:18 smithi129 ceph-mon[42066]: osdmap e381: 8 total, 8 up, 8 in 2024-03-11T20:59:18.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:18 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:59:18.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:18 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:18.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:18 smithi157 ceph-mon[33359]: pgmap v529: 292 pgs: 292 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-11T20:59:18.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:18 smithi157 ceph-mon[33359]: osdmap e381: 8 total, 8 up, 8 in 2024-03-11T20:59:18.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:18 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:59:18.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:18 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:19.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:19 smithi129 ceph-mon[30184]: osdmap e382: 8 total, 8 up, 8 in 2024-03-11T20:59:19.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:19.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:19.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:19 smithi129 ceph-mon[42066]: osdmap e382: 8 total, 8 up, 8 in 2024-03-11T20:59:19.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:19.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:19.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:19 smithi157 ceph-mon[33359]: osdmap e382: 8 total, 8 up, 8 in 2024-03-11T20:59:19.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:19.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:20.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[42066]: pgmap v532: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T20:59:20.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:59:20.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:20.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[42066]: osdmap e383: 8 total, 8 up, 8 in 2024-03-11T20:59:20.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T20:59:20.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:20.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:20.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-59"}]: dispatch 2024-03-11T20:59:20.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-59"}]: dispatch 2024-03-11T20:59:20.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[30184]: pgmap v532: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T20:59:20.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:59:20.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:20.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[30184]: osdmap e383: 8 total, 8 up, 8 in 2024-03-11T20:59:20.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T20:59:20.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:20.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:20.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-59"}]: dispatch 2024-03-11T20:59:20.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-59"}]: dispatch 2024-03-11T20:59:20.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:20 smithi157 ceph-mon[33359]: pgmap v532: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T20:59:20.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:20 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:59:20.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:20.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:20 smithi157 ceph-mon[33359]: osdmap e383: 8 total, 8 up, 8 in 2024-03-11T20:59:20.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T20:59:20.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:20.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:20.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-59"}]: dispatch 2024-03-11T20:59:20.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-59"}]: dispatch 2024-03-11T20:59:21.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:21 smithi129 ceph-mon[30184]: osdmap e384: 8 total, 8 up, 8 in 2024-03-11T20:59:21.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:21 smithi129 ceph-mon[42066]: osdmap e384: 8 total, 8 up, 8 in 2024-03-11T20:59:21.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:21 smithi157 ceph-mon[33359]: osdmap e384: 8 total, 8 up, 8 in 2024-03-11T20:59:22.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:22 smithi129 ceph-mon[30184]: pgmap v535: 260 pgs: 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:59:22.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:22 smithi129 ceph-mon[30184]: osdmap e385: 8 total, 8 up, 8 in 2024-03-11T20:59:22.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:22.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:22 smithi129 ceph-mon[42066]: pgmap v535: 260 pgs: 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:59:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:22 smithi129 ceph-mon[42066]: osdmap e385: 8 total, 8 up, 8 in 2024-03-11T20:59:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:22.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:22 smithi157 ceph-mon[33359]: pgmap v535: 260 pgs: 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:59:22.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:22 smithi157 ceph-mon[33359]: osdmap e385: 8 total, 8 up, 8 in 2024-03-11T20:59:22.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:22.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:23 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:23 smithi129 ceph-mon[30184]: osdmap e386: 8 total, 8 up, 8 in 2024-03-11T20:59:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T20:59:23.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T20:59:23.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:23 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T20:59:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:23 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:23 smithi129 ceph-mon[42066]: osdmap e386: 8 total, 8 up, 8 in 2024-03-11T20:59:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T20:59:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T20:59:23.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:23 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T20:59:23.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:59:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:59:23.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:23 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:23.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:23 smithi157 ceph-mon[33359]: osdmap e386: 8 total, 8 up, 8 in 2024-03-11T20:59:23.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T20:59:23.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T20:59:23.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:23 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T20:59:24.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:24 smithi129 ceph-mon[30184]: pgmap v538: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T20:59:24.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:24 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:24.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T20:59:24.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:24 smithi129 ceph-mon[30184]: osdmap e387: 8 total, 8 up, 8 in 2024-03-11T20:59:24.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:24.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:24.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-61"}]: dispatch 2024-03-11T20:59:24.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-61"}]: dispatch 2024-03-11T20:59:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:24 smithi129 ceph-mon[42066]: pgmap v538: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T20:59:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:24 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T20:59:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:24 smithi129 ceph-mon[42066]: osdmap e387: 8 total, 8 up, 8 in 2024-03-11T20:59:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-61"}]: dispatch 2024-03-11T20:59:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-61"}]: dispatch 2024-03-11T20:59:24.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:24 smithi157 ceph-mon[33359]: pgmap v538: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T20:59:24.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:24 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:24.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T20:59:24.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:24 smithi157 ceph-mon[33359]: osdmap e387: 8 total, 8 up, 8 in 2024-03-11T20:59:24.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:24.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T20:59:24.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-61"}]: dispatch 2024-03-11T20:59:24.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-61"}]: dispatch 2024-03-11T20:59:25.880 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:25 smithi129 ceph-mon[30184]: osdmap e388: 8 total, 8 up, 8 in 2024-03-11T20:59:25.880 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:25 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:59:25.880 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:25 smithi129 ceph-mon[42066]: osdmap e388: 8 total, 8 up, 8 in 2024-03-11T20:59:25.880 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:25 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:59:25.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:25 smithi157 ceph-mon[33359]: osdmap e388: 8 total, 8 up, 8 in 2024-03-11T20:59:25.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:25 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T20:59:26.229 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:59:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:59:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:59:26.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:26 smithi129 ceph-mon[30184]: pgmap v541: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T20:59:26.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:26 smithi129 ceph-mon[30184]: osdmap e389: 8 total, 8 up, 8 in 2024-03-11T20:59:26.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:26 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:26.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:26 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:26.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:26 smithi129 ceph-mon[42066]: pgmap v541: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T20:59:26.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:26 smithi129 ceph-mon[42066]: osdmap e389: 8 total, 8 up, 8 in 2024-03-11T20:59:26.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:26 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:26.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:26 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:26.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:26 smithi157 ceph-mon[33359]: pgmap v541: 260 pgs: 260 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T20:59:26.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:26 smithi157 ceph-mon[33359]: osdmap e389: 8 total, 8 up, 8 in 2024-03-11T20:59:26.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:26 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:26.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:26 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T20:59:27.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:27 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:27.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:27 smithi129 ceph-mon[30184]: osdmap e390: 8 total, 8 up, 8 in 2024-03-11T20:59:27.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T20:59:27.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T20:59:27.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:27 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T20:59:27.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:27 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:27.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:27 smithi129 ceph-mon[42066]: osdmap e390: 8 total, 8 up, 8 in 2024-03-11T20:59:27.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T20:59:27.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T20:59:27.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:27 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T20:59:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:27 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T20:59:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:27 smithi157 ceph-mon[33359]: osdmap e390: 8 total, 8 up, 8 in 2024-03-11T20:59:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T20:59:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T20:59:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:27 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T20:59:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:28 smithi129 ceph-mon[30184]: pgmap v544: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:59:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T20:59:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:28 smithi129 ceph-mon[30184]: osdmap e391: 8 total, 8 up, 8 in 2024-03-11T20:59:28.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:28 smithi129 ceph-mon[42066]: pgmap v544: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:59:28.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T20:59:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:28 smithi129 ceph-mon[42066]: osdmap e391: 8 total, 8 up, 8 in 2024-03-11T20:59:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:28 smithi157 ceph-mon[33359]: pgmap v544: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:59:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T20:59:28.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:28 smithi157 ceph-mon[33359]: osdmap e391: 8 total, 8 up, 8 in 2024-03-11T20:59:29.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:29 smithi129 ceph-mon[30184]: osdmap e392: 8 total, 8 up, 8 in 2024-03-11T20:59:29.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:29 smithi129 ceph-mon[42066]: osdmap e392: 8 total, 8 up, 8 in 2024-03-11T20:59:29.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:29 smithi157 ceph-mon[33359]: osdmap e392: 8 total, 8 up, 8 in 2024-03-11T20:59:30.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:30 smithi129 ceph-mon[30184]: pgmap v547: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:59:30.982 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:30 smithi129 ceph-mon[30184]: osdmap e393: 8 total, 8 up, 8 in 2024-03-11T20:59:30.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:30 smithi129 ceph-mon[42066]: pgmap v547: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:59:30.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:30 smithi129 ceph-mon[42066]: osdmap e393: 8 total, 8 up, 8 in 2024-03-11T20:59:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:30 smithi157 ceph-mon[33359]: pgmap v547: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T20:59:30.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:30 smithi157 ceph-mon[33359]: osdmap e393: 8 total, 8 up, 8 in 2024-03-11T20:59:31.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:31 smithi129 ceph-mon[30184]: osdmap e394: 8 total, 8 up, 8 in 2024-03-11T20:59:31.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:31 smithi129 ceph-mon[42066]: osdmap e394: 8 total, 8 up, 8 in 2024-03-11T20:59:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:31 smithi157 ceph-mon[33359]: osdmap e394: 8 total, 8 up, 8 in 2024-03-11T20:59:32.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:32 smithi129 ceph-mon[30184]: pgmap v550: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:59:32.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:32 smithi129 ceph-mon[30184]: osdmap e395: 8 total, 8 up, 8 in 2024-03-11T20:59:32.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:32 smithi129 ceph-mon[42066]: pgmap v550: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:59:32.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:32 smithi129 ceph-mon[42066]: osdmap e395: 8 total, 8 up, 8 in 2024-03-11T20:59:32.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:32 smithi157 ceph-mon[33359]: pgmap v550: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-03-11T20:59:32.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:32 smithi157 ceph-mon[33359]: osdmap e395: 8 total, 8 up, 8 in 2024-03-11T20:59:33.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:33 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:33.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:33 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:33.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:33 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:33.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:59:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:59:34.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:34 smithi129 ceph-mon[30184]: pgmap v552: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2.6 KiB/s wr, 7 op/s 2024-03-11T20:59:34.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:34 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:34.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:34 smithi129 ceph-mon[42066]: pgmap v552: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2.6 KiB/s wr, 7 op/s 2024-03-11T20:59:34.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:34 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:34.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:34 smithi157 ceph-mon[33359]: pgmap v552: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2.6 KiB/s wr, 7 op/s 2024-03-11T20:59:34.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:34 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:36.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:59:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:59:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:59:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:36 smithi129 ceph-mon[30184]: pgmap v553: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-11T20:59:36.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:36 smithi129 ceph-mon[42066]: pgmap v553: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-11T20:59:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:36 smithi157 ceph-mon[33359]: pgmap v553: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-11T20:59:38.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:38 smithi129 ceph-mon[30184]: pgmap v554: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T20:59:38.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:38 smithi129 ceph-mon[42066]: pgmap v554: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T20:59:38.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:38 smithi157 ceph-mon[33359]: pgmap v554: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T20:59:40.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:40 smithi129 ceph-mon[30184]: pgmap v555: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-11T20:59:40.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:40 smithi129 ceph-mon[30184]: osdmap e396: 8 total, 8 up, 8 in 2024-03-11T20:59:40.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:40 smithi129 ceph-mon[42066]: pgmap v555: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-11T20:59:40.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:40 smithi129 ceph-mon[42066]: osdmap e396: 8 total, 8 up, 8 in 2024-03-11T20:59:40.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:40 smithi157 ceph-mon[33359]: pgmap v555: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-11T20:59:40.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:40 smithi157 ceph-mon[33359]: osdmap e396: 8 total, 8 up, 8 in 2024-03-11T20:59:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:42 smithi129 ceph-mon[30184]: pgmap v557: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-11T20:59:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:42 smithi129 ceph-mon[30184]: osdmap e397: 8 total, 8 up, 8 in 2024-03-11T20:59:42.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:42 smithi129 ceph-mon[42066]: pgmap v557: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-11T20:59:42.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:42 smithi129 ceph-mon[42066]: osdmap e397: 8 total, 8 up, 8 in 2024-03-11T20:59:42.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:42 smithi157 ceph-mon[33359]: pgmap v557: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-11T20:59:42.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:42 smithi157 ceph-mon[33359]: osdmap e397: 8 total, 8 up, 8 in 2024-03-11T20:59:43.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:59:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:59:44.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:44 smithi129 ceph-mon[30184]: pgmap v559: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:44.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:44 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:44.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:44 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:59:44.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:44 smithi129 ceph-mon[42066]: pgmap v559: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:44.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:44 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:44.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:44 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:59:44.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:44 smithi157 ceph-mon[33359]: pgmap v559: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:44.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:44 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:44.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:44 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T20:59:45.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:45 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:59:45.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:45 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:59:45.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:45 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:59:45.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:45 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:59:45.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:45 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:59:45.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:45 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:59:45.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:45 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T20:59:45.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:45 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T20:59:45.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:45 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T20:59:46.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:59:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:59:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:59:46.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:46 smithi129 ceph-mon[30184]: pgmap v560: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T20:59:46.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:46 smithi129 ceph-mon[42066]: pgmap v560: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T20:59:46.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:46 smithi157 ceph-mon[33359]: pgmap v560: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T20:59:48.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:48 smithi129 ceph-mon[30184]: pgmap v561: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:48.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:48 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:48.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:48 smithi129 ceph-mon[42066]: pgmap v561: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:48.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:48 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:48.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:48 smithi157 ceph-mon[33359]: pgmap v561: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:48.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:48 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T20:59:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:50 smithi129 ceph-mon[30184]: pgmap v562: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:50.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:50 smithi129 ceph-mon[30184]: osdmap e398: 8 total, 8 up, 8 in 2024-03-11T20:59:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:50 smithi129 ceph-mon[42066]: pgmap v562: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:50.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:50 smithi129 ceph-mon[42066]: osdmap e398: 8 total, 8 up, 8 in 2024-03-11T20:59:50.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:50 smithi157 ceph-mon[33359]: pgmap v562: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:50.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:50 smithi157 ceph-mon[33359]: osdmap e398: 8 total, 8 up, 8 in 2024-03-11T20:59:52.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:52 smithi129 ceph-mon[30184]: pgmap v564: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:52.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:52 smithi129 ceph-mon[30184]: osdmap e399: 8 total, 8 up, 8 in 2024-03-11T20:59:52.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:52 smithi129 ceph-mon[42066]: pgmap v564: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:52.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:52 smithi129 ceph-mon[42066]: osdmap e399: 8 total, 8 up, 8 in 2024-03-11T20:59:52.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:52 smithi157 ceph-mon[33359]: pgmap v564: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:52.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:52 smithi157 ceph-mon[33359]: osdmap e399: 8 total, 8 up, 8 in 2024-03-11T20:59:53.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 20:59:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T20:59:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:54 smithi129 ceph-mon[30184]: pgmap v566: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:54.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:54 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:54.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:54 smithi129 ceph-mon[42066]: pgmap v566: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:54.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:54 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:54.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:54 smithi157 ceph-mon[33359]: pgmap v566: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T20:59:54.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:54 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T20:59:56.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 20:59:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:20:59:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T20:59:56.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:56 smithi129 ceph-mon[30184]: pgmap v567: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T20:59:56.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:56 smithi129 ceph-mon[42066]: pgmap v567: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T20:59:56.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:56 smithi157 ceph-mon[33359]: pgmap v567: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T20:59:58.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 20:59:58 smithi129 ceph-mon[30184]: pgmap v568: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T20:59:58.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 20:59:58 smithi129 ceph-mon[42066]: pgmap v568: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T20:59:58.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 20:59:58 smithi157 ceph-mon[33359]: pgmap v568: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:00:00.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:00 smithi129 ceph-mon[30184]: pgmap v569: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:00:00.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:00 smithi129 ceph-mon[30184]: osdmap e400: 8 total, 8 up, 8 in 2024-03-11T21:00:00.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:00 smithi129 ceph-mon[30184]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-11T21:00:00.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:00 smithi129 ceph-mon[30184]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-11T21:00:00.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:00 smithi129 ceph-mon[30184]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-11T21:00:00.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:00 smithi129 ceph-mon[30184]: application not enabled on pool 'poolrace.1804289383' 2024-03-11T21:00:00.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:00 smithi129 ceph-mon[30184]: application not enabled on pool 'poolrace2.846930886' 2024-03-11T21:00:00.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:00 smithi129 ceph-mon[30184]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-11T21:00:00.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:00 smithi129 ceph-mon[42066]: pgmap v569: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:00:00.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:00 smithi129 ceph-mon[42066]: osdmap e400: 8 total, 8 up, 8 in 2024-03-11T21:00:00.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:00 smithi129 ceph-mon[42066]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-11T21:00:00.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:00 smithi129 ceph-mon[42066]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-11T21:00:00.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:00 smithi129 ceph-mon[42066]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-11T21:00:00.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:00 smithi129 ceph-mon[42066]: application not enabled on pool 'poolrace.1804289383' 2024-03-11T21:00:00.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:00 smithi129 ceph-mon[42066]: application not enabled on pool 'poolrace2.846930886' 2024-03-11T21:00:00.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:00 smithi129 ceph-mon[42066]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-11T21:00:00.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:00 smithi157 ceph-mon[33359]: pgmap v569: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:00:00.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:00 smithi157 ceph-mon[33359]: osdmap e400: 8 total, 8 up, 8 in 2024-03-11T21:00:00.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:00 smithi157 ceph-mon[33359]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-11T21:00:00.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:00 smithi157 ceph-mon[33359]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-11T21:00:00.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:00 smithi157 ceph-mon[33359]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-11T21:00:00.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:00 smithi157 ceph-mon[33359]: application not enabled on pool 'poolrace.1804289383' 2024-03-11T21:00:00.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:00 smithi157 ceph-mon[33359]: application not enabled on pool 'poolrace2.846930886' 2024-03-11T21:00:00.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:00 smithi157 ceph-mon[33359]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-11T21:00:02.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:02 smithi129 ceph-mon[30184]: pgmap v571: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:00:02.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:02 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:02.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:02 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:02.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:02 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-63"}]: dispatch 2024-03-11T21:00:02.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:02 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-63"}]: dispatch 2024-03-11T21:00:02.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:02 smithi129 ceph-mon[42066]: pgmap v571: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:00:02.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:02 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:02.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:02 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:02.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:02 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-63"}]: dispatch 2024-03-11T21:00:02.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:02 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-63"}]: dispatch 2024-03-11T21:00:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:02 smithi157 ceph-mon[33359]: pgmap v571: 292 pgs: 292 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:00:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:02 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:02 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:02.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:02 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-63"}]: dispatch 2024-03-11T21:00:02.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:02 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-63"}]: dispatch 2024-03-11T21:00:03.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:03 smithi129 ceph-mon[30184]: osdmap e401: 8 total, 8 up, 8 in 2024-03-11T21:00:03.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:03 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:03.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:03 smithi129 ceph-mon[42066]: osdmap e401: 8 total, 8 up, 8 in 2024-03-11T21:00:03.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:03 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:03.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:00:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:00:03.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:03 smithi157 ceph-mon[33359]: osdmap e401: 8 total, 8 up, 8 in 2024-03-11T21:00:03.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:03 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:05.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:04 smithi129 ceph-mon[30184]: pgmap v573: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:05.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:04 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:05.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:04 smithi129 ceph-mon[30184]: osdmap e402: 8 total, 8 up, 8 in 2024-03-11T21:00:05.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:05.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:05.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:04 smithi129 ceph-mon[42066]: pgmap v573: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:05.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:04 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:05.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:04 smithi129 ceph-mon[42066]: osdmap e402: 8 total, 8 up, 8 in 2024-03-11T21:00:05.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:05.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:05.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:04 smithi157 ceph-mon[33359]: pgmap v573: 260 pgs: 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:05.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:04 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:05.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:04 smithi157 ceph-mon[33359]: osdmap e402: 8 total, 8 up, 8 in 2024-03-11T21:00:05.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:05.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:06.106 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:05 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:00:06.106 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:05 smithi129 ceph-mon[30184]: osdmap e403: 8 total, 8 up, 8 in 2024-03-11T21:00:06.106 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:00:06.106 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:00:06.106 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:05 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:00:06.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:05 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:00:06.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:05 smithi129 ceph-mon[42066]: osdmap e403: 8 total, 8 up, 8 in 2024-03-11T21:00:06.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:00:06.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:00:06.107 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:05 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:00:06.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:05 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:00:06.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:05 smithi157 ceph-mon[33359]: osdmap e403: 8 total, 8 up, 8 in 2024-03-11T21:00:06.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:00:06.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:00:06.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:05 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:00:06.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:00:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:00:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:00:07.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:06 smithi129 ceph-mon[30184]: pgmap v576: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:07.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:06 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:00:07.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:06 smithi129 ceph-mon[30184]: osdmap e404: 8 total, 8 up, 8 in 2024-03-11T21:00:07.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:06 smithi129 ceph-mon[42066]: pgmap v576: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:07.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:06 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:00:07.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:06 smithi129 ceph-mon[42066]: osdmap e404: 8 total, 8 up, 8 in 2024-03-11T21:00:07.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:06 smithi157 ceph-mon[33359]: pgmap v576: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:07.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:06 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:00:07.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:06 smithi157 ceph-mon[33359]: osdmap e404: 8 total, 8 up, 8 in 2024-03-11T21:00:08.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:07 smithi129 ceph-mon[30184]: osdmap e405: 8 total, 8 up, 8 in 2024-03-11T21:00:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:07 smithi129 ceph-mon[42066]: osdmap e405: 8 total, 8 up, 8 in 2024-03-11T21:00:08.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:07 smithi157 ceph-mon[33359]: osdmap e405: 8 total, 8 up, 8 in 2024-03-11T21:00:09.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:08 smithi129 ceph-mon[30184]: pgmap v579: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-11T21:00:09.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:08 smithi129 ceph-mon[30184]: osdmap e406: 8 total, 8 up, 8 in 2024-03-11T21:00:09.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:08 smithi129 ceph-mon[42066]: pgmap v579: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-11T21:00:09.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:08 smithi129 ceph-mon[42066]: osdmap e406: 8 total, 8 up, 8 in 2024-03-11T21:00:09.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:08 smithi157 ceph-mon[33359]: pgmap v579: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-11T21:00:09.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:08 smithi157 ceph-mon[33359]: osdmap e406: 8 total, 8 up, 8 in 2024-03-11T21:00:10.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:09 smithi129 ceph-mon[30184]: osdmap e407: 8 total, 8 up, 8 in 2024-03-11T21:00:10.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:09 smithi129 ceph-mon[30184]: pgmap v582: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-11T21:00:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:09 smithi129 ceph-mon[42066]: osdmap e407: 8 total, 8 up, 8 in 2024-03-11T21:00:10.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:09 smithi129 ceph-mon[42066]: pgmap v582: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-11T21:00:10.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:09 smithi157 ceph-mon[33359]: osdmap e407: 8 total, 8 up, 8 in 2024-03-11T21:00:10.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:09 smithi157 ceph-mon[33359]: pgmap v582: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-11T21:00:12.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:11 smithi129 ceph-mon[30184]: pgmap v583: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-11T21:00:12.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:11 smithi129 ceph-mon[42066]: pgmap v583: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-11T21:00:12.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:11 smithi157 ceph-mon[33359]: pgmap v583: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-11T21:00:13.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:00:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:00:14.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:13 smithi129 ceph-mon[30184]: pgmap v584: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-11T21:00:14.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:13 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:13 smithi129 ceph-mon[42066]: pgmap v584: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-11T21:00:14.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:13 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:14.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:13 smithi157 ceph-mon[33359]: pgmap v584: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-11T21:00:14.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:13 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:16.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:15 smithi129 ceph-mon[30184]: osdmap e408: 8 total, 8 up, 8 in 2024-03-11T21:00:16.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:15 smithi129 ceph-mon[30184]: pgmap v586: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 287 B/s wr, 3 op/s 2024-03-11T21:00:16.229 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:00:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:00:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:00:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:15 smithi129 ceph-mon[42066]: osdmap e408: 8 total, 8 up, 8 in 2024-03-11T21:00:16.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:15 smithi129 ceph-mon[42066]: pgmap v586: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 287 B/s wr, 3 op/s 2024-03-11T21:00:16.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:15 smithi157 ceph-mon[33359]: osdmap e408: 8 total, 8 up, 8 in 2024-03-11T21:00:16.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:15 smithi157 ceph-mon[33359]: pgmap v586: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 287 B/s wr, 3 op/s 2024-03-11T21:00:18.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:17 smithi129 ceph-mon[42066]: pgmap v587: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 252 B/s wr, 3 op/s 2024-03-11T21:00:18.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:17 smithi129 ceph-mon[30184]: pgmap v587: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 252 B/s wr, 3 op/s 2024-03-11T21:00:18.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:17 smithi157 ceph-mon[33359]: pgmap v587: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 252 B/s wr, 3 op/s 2024-03-11T21:00:19.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:18 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:19.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:19.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:19.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-65"}]: dispatch 2024-03-11T21:00:19.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-65"}]: dispatch 2024-03-11T21:00:19.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:18 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:19.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:19.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:19.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-65"}]: dispatch 2024-03-11T21:00:19.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-65"}]: dispatch 2024-03-11T21:00:19.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:18 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:19.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:19.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:19.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-65"}]: dispatch 2024-03-11T21:00:19.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-65"}]: dispatch 2024-03-11T21:00:20.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:19 smithi129 ceph-mon[42066]: pgmap v588: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-11T21:00:20.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:19 smithi129 ceph-mon[42066]: osdmap e409: 8 total, 8 up, 8 in 2024-03-11T21:00:20.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:19 smithi129 ceph-mon[30184]: pgmap v588: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-11T21:00:20.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:19 smithi129 ceph-mon[30184]: osdmap e409: 8 total, 8 up, 8 in 2024-03-11T21:00:20.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:19 smithi157 ceph-mon[33359]: pgmap v588: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-11T21:00:20.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:19 smithi157 ceph-mon[33359]: osdmap e409: 8 total, 8 up, 8 in 2024-03-11T21:00:21.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:20 smithi129 ceph-mon[30184]: osdmap e410: 8 total, 8 up, 8 in 2024-03-11T21:00:21.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:21.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:21.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:20 smithi129 ceph-mon[42066]: osdmap e410: 8 total, 8 up, 8 in 2024-03-11T21:00:21.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:21.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:21.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:20 smithi157 ceph-mon[33359]: osdmap e410: 8 total, 8 up, 8 in 2024-03-11T21:00:21.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:21.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:22.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:21 smithi129 ceph-mon[42066]: pgmap v591: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:22.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:21 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:00:22.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:21 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:00:22.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:21 smithi129 ceph-mon[42066]: osdmap e411: 8 total, 8 up, 8 in 2024-03-11T21:00:22.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:00:22.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:21 smithi129 ceph-mon[30184]: pgmap v591: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:22.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:21 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:00:22.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:21 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:00:22.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:21 smithi129 ceph-mon[30184]: osdmap e411: 8 total, 8 up, 8 in 2024-03-11T21:00:22.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:00:22.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:21 smithi157 ceph-mon[33359]: pgmap v591: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:22.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:21 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:00:22.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:21 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:00:22.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:21 smithi157 ceph-mon[33359]: osdmap e411: 8 total, 8 up, 8 in 2024-03-11T21:00:22.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:00:23.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:22 smithi129 ceph-mon[30184]: osdmap e412: 8 total, 8 up, 8 in 2024-03-11T21:00:23.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:23.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:23.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-67"}]: dispatch 2024-03-11T21:00:23.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-67"}]: dispatch 2024-03-11T21:00:23.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:22 smithi129 ceph-mon[42066]: osdmap e412: 8 total, 8 up, 8 in 2024-03-11T21:00:23.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:23.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:23.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-67"}]: dispatch 2024-03-11T21:00:23.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-67"}]: dispatch 2024-03-11T21:00:23.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:22 smithi157 ceph-mon[33359]: osdmap e412: 8 total, 8 up, 8 in 2024-03-11T21:00:23.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-67"}]: dispatch 2024-03-11T21:00:23.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-67"}]: dispatch 2024-03-11T21:00:23.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:00:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:00:24.154 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a46880 2024-03-11T21:00:24.154 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a460e0 2024-03-11T21:00:24.154 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a46210 2024-03-11T21:00:24.154 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a46340 2024-03-11T21:00:24.155 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a469d0 2024-03-11T21:00:24.155 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a46ea0 2024-03-11T21:00:24.155 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a46fd0 2024-03-11T21:00:24.155 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a45130 2024-03-11T21:00:24.155 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a45260 2024-03-11T21:00:24.155 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a45390 2024-03-11T21:00:24.155 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a47100 2024-03-11T21:00:24.155 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a475d0 2024-03-11T21:00:24.155 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a47720 2024-03-11T21:00:24.155 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a46b00 2024-03-11T21:00:24.155 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a46c30 2024-03-11T21:00:24.155 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a46d60 2024-03-11T21:00:24.156 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a47850 2024-03-11T21:00:24.156 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a47d20 2024-03-11T21:00:24.156 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a47e50 2024-03-11T21:00:24.156 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a48410 2024-03-11T21:00:24.156 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a47980 2024-03-11T21:00:24.156 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a47ab0 2024-03-11T21:00:24.156 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a47c00 2024-03-11T21:00:24.156 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a48560 2024-03-11T21:00:24.156 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a47f60 2024-03-11T21:00:24.156 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a48090 2024-03-11T21:00:24.156 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a481a0 2024-03-11T21:00:24.156 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a482f0 2024-03-11T21:00:24.156 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a48b40 2024-03-11T21:00:24.156 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a48690 2024-03-11T21:00:24.157 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a487c0 2024-03-11T21:00:24.157 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a488d0 2024-03-11T21:00:24.157 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a48a20 2024-03-11T21:00:24.157 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a49140 2024-03-11T21:00:24.157 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a48c90 2024-03-11T21:00:24.157 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a48de0 2024-03-11T21:00:24.157 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a48f30 2024-03-11T21:00:24.157 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a492b0 2024-03-11T21:00:24.157 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a49760 2024-03-11T21:00:24.157 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a49890 2024-03-11T21:00:24.157 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a47230 2024-03-11T21:00:24.157 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a47360 2024-03-11T21:00:24.157 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a47490 2024-03-11T21:00:24.157 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a499a0 2024-03-11T21:00:24.157 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a49e70 2024-03-11T21:00:24.157 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a49fc0 2024-03-11T21:00:24.158 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a493c0 2024-03-11T21:00:24.158 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a494f0 2024-03-11T21:00:24.158 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a49620 2024-03-11T21:00:24.158 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4a0f0 2024-03-11T21:00:24.158 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4a5c0 2024-03-11T21:00:24.158 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4a710 2024-03-11T21:00:24.158 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a49ad0 2024-03-11T21:00:24.158 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a49c00 2024-03-11T21:00:24.158 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a49d30 2024-03-11T21:00:24.158 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4a840 2024-03-11T21:00:24.158 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4ad10 2024-03-11T21:00:24.158 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4ae60 2024-03-11T21:00:24.158 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4a220 2024-03-11T21:00:24.159 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4a350 2024-03-11T21:00:24.159 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4a460 2024-03-11T21:00:24.159 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4af90 2024-03-11T21:00:24.159 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4b460 2024-03-11T21:00:24.159 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4b590 2024-03-11T21:00:24.159 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4bb70 2024-03-11T21:00:24.159 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4b0c0 2024-03-11T21:00:24.159 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4b1f0 2024-03-11T21:00:24.159 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4b320 2024-03-11T21:00:24.159 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4bcc0 2024-03-11T21:00:24.159 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4b6c0 2024-03-11T21:00:24.159 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4b810 2024-03-11T21:00:24.159 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4b920 2024-03-11T21:00:24.160 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4bdf0 2024-03-11T21:00:24.160 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4c310 2024-03-11T21:00:24.160 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4c490 2024-03-11T21:00:24.160 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4a970 2024-03-11T21:00:24.160 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4aac0 2024-03-11T21:00:24.160 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4abf0 2024-03-11T21:00:24.160 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4c5a0 2024-03-11T21:00:24.160 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4ca50 2024-03-11T21:00:24.160 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4cba0 2024-03-11T21:00:24.160 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4bf20 2024-03-11T21:00:24.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:23 smithi129 ceph-mon[30184]: pgmap v594: 292 pgs: 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:24.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:23 smithi129 ceph-mon[30184]: osdmap e413: 8 total, 8 up, 8 in 2024-03-11T21:00:24.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:23 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:24.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:23 smithi129 ceph-mon[42066]: pgmap v594: 292 pgs: 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:24.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:23 smithi129 ceph-mon[42066]: osdmap e413: 8 total, 8 up, 8 in 2024-03-11T21:00:24.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:23 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:24.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:23 smithi157 ceph-mon[33359]: pgmap v594: 292 pgs: 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:23 smithi157 ceph-mon[33359]: osdmap e413: 8 total, 8 up, 8 in 2024-03-11T21:00:24.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:23 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:24 smithi129 ceph-mon[30184]: osdmap e414: 8 total, 8 up, 8 in 2024-03-11T21:00:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:25.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:25.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:24 smithi129 ceph-mon[42066]: osdmap e414: 8 total, 8 up, 8 in 2024-03-11T21:00:25.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:25.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:25.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:24 smithi157 ceph-mon[33359]: osdmap e414: 8 total, 8 up, 8 in 2024-03-11T21:00:25.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:25.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:26.229 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:00:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:00:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:00:26.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:25 smithi129 ceph-mon[42066]: pgmap v597: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:00:26.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:00:26.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:25 smithi129 ceph-mon[42066]: osdmap e415: 8 total, 8 up, 8 in 2024-03-11T21:00:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:00:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-69"}]: dispatch 2024-03-11T21:00:26.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-69"}]: dispatch 2024-03-11T21:00:26.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:25 smithi129 ceph-mon[30184]: pgmap v597: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:00:26.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:00:26.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:25 smithi129 ceph-mon[30184]: osdmap e415: 8 total, 8 up, 8 in 2024-03-11T21:00:26.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:00:26.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:26.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:26.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-69"}]: dispatch 2024-03-11T21:00:26.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-69"}]: dispatch 2024-03-11T21:00:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:25 smithi157 ceph-mon[33359]: pgmap v597: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:00:26.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:00:26.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:25 smithi157 ceph-mon[33359]: osdmap e415: 8 total, 8 up, 8 in 2024-03-11T21:00:26.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:00:26.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:26.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:26.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-69"}]: dispatch 2024-03-11T21:00:26.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-69"}]: dispatch 2024-03-11T21:00:27.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:26 smithi129 ceph-mon[30184]: osdmap e416: 8 total, 8 up, 8 in 2024-03-11T21:00:27.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:26 smithi129 ceph-mon[42066]: osdmap e416: 8 total, 8 up, 8 in 2024-03-11T21:00:27.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:26 smithi157 ceph-mon[33359]: osdmap e416: 8 total, 8 up, 8 in 2024-03-11T21:00:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:27 smithi129 ceph-mon[30184]: pgmap v600: 260 pgs: 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:00:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:27 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:00:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:27 smithi129 ceph-mon[30184]: osdmap e417: 8 total, 8 up, 8 in 2024-03-11T21:00:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:28.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:27 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:28.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:27 smithi129 ceph-mon[42066]: pgmap v600: 260 pgs: 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:00:28.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:27 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:00:28.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:27 smithi129 ceph-mon[42066]: osdmap e417: 8 total, 8 up, 8 in 2024-03-11T21:00:28.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:28.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:27 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:27 smithi157 ceph-mon[33359]: pgmap v600: 260 pgs: 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:00:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:27 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:00:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:27 smithi157 ceph-mon[33359]: osdmap e417: 8 total, 8 up, 8 in 2024-03-11T21:00:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:28.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:27 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:00:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:29 smithi129 ceph-mon[30184]: osdmap e418: 8 total, 8 up, 8 in 2024-03-11T21:00:29.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:00:29.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:00:29.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:00:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:00:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:29 smithi129 ceph-mon[42066]: osdmap e418: 8 total, 8 up, 8 in 2024-03-11T21:00:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:00:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:00:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:00:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:00:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:29 smithi157 ceph-mon[33359]: osdmap e418: 8 total, 8 up, 8 in 2024-03-11T21:00:29.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:00:29.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:00:29.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:00:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:30 smithi129 ceph-mon[30184]: pgmap v603: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:00:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:30 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:00:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:30 smithi129 ceph-mon[30184]: osdmap e419: 8 total, 8 up, 8 in 2024-03-11T21:00:30.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:30 smithi129 ceph-mon[42066]: pgmap v603: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:00:30.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:30 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:00:30.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:30 smithi129 ceph-mon[42066]: osdmap e419: 8 total, 8 up, 8 in 2024-03-11T21:00:30.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:30 smithi157 ceph-mon[33359]: pgmap v603: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:00:30.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:30 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:00:30.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:30 smithi157 ceph-mon[33359]: osdmap e419: 8 total, 8 up, 8 in 2024-03-11T21:00:31.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:31 smithi129 ceph-mon[30184]: osdmap e420: 8 total, 8 up, 8 in 2024-03-11T21:00:31.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:31 smithi129 ceph-mon[42066]: osdmap e420: 8 total, 8 up, 8 in 2024-03-11T21:00:31.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:31 smithi157 ceph-mon[33359]: osdmap e420: 8 total, 8 up, 8 in 2024-03-11T21:00:32.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:32 smithi129 ceph-mon[30184]: pgmap v606: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:32.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:32 smithi129 ceph-mon[42066]: pgmap v606: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:32.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:32 smithi157 ceph-mon[33359]: pgmap v606: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:33.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:00:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:00:34.460 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:34 smithi157 ceph-mon[33359]: pgmap v607: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-11T21:00:34.460 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:34 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:34.461 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:34 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:34.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:34 smithi129 ceph-mon[30184]: pgmap v607: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-11T21:00:34.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:34 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:34.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:34 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:34.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:34 smithi129 ceph-mon[42066]: pgmap v607: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-11T21:00:34.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:34 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:34.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:34 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:36.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:36 smithi129 ceph-mon[30184]: pgmap v608: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 1.9 KiB/s wr, 4 op/s 2024-03-11T21:00:36.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:36 smithi129 ceph-mon[42066]: pgmap v608: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 1.9 KiB/s wr, 4 op/s 2024-03-11T21:00:36.480 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:00:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:00:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:00:36.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:36 smithi157 ceph-mon[33359]: pgmap v608: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 1.9 KiB/s wr, 4 op/s 2024-03-11T21:00:38.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:38 smithi129 ceph-mon[30184]: pgmap v609: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-11T21:00:38.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:38 smithi129 ceph-mon[42066]: pgmap v609: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-11T21:00:38.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:38 smithi157 ceph-mon[33359]: pgmap v609: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-11T21:00:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:40 smithi129 ceph-mon[30184]: pgmap v610: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-11T21:00:40.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:40 smithi129 ceph-mon[42066]: pgmap v610: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-11T21:00:40.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:40 smithi157 ceph-mon[33359]: pgmap v610: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-11T21:00:42.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:42 smithi129 ceph-mon[30184]: pgmap v611: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-11T21:00:42.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:42 smithi129 ceph-mon[42066]: pgmap v611: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-11T21:00:42.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:42 smithi157 ceph-mon[33359]: pgmap v611: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-11T21:00:43.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:00:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:00:44.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:44 smithi129 ceph-mon[42066]: pgmap v612: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-11T21:00:44.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:44 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:44.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:44 smithi129 ceph-mon[30184]: pgmap v612: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-11T21:00:44.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:44 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:44.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:44 smithi157 ceph-mon[33359]: pgmap v612: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-11T21:00:44.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:44 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:46.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:00:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:00:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:00:46.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:46 smithi129 ceph-mon[30184]: pgmap v613: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:00:46.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:46 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:00:46.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:46 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:00:46.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:46 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:00:46.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:46 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:00:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:46 smithi129 ceph-mon[42066]: pgmap v613: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:00:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:46 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:00:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:46 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:00:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:46 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:00:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:46 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:00:46.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:46 smithi157 ceph-mon[33359]: pgmap v613: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:00:46.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:46 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:00:46.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:46 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:00:46.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:46 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:00:46.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:46 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:00:48.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:48 smithi129 ceph-mon[30184]: pgmap v614: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:00:48.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:48 smithi129 ceph-mon[42066]: pgmap v614: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:00:48.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:48 smithi157 ceph-mon[33359]: pgmap v614: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:00:49.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:49 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:49.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:49 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:49.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:49 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:00:50.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:50 smithi129 ceph-mon[30184]: pgmap v615: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:00:50.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:50 smithi129 ceph-mon[42066]: pgmap v615: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:00:50.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:50 smithi157 ceph-mon[33359]: pgmap v615: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:00:51.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:51.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:51 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:51.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-71"}]: dispatch 2024-03-11T21:00:51.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:51 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-71"}]: dispatch 2024-03-11T21:00:51.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:51.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:51 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:51.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-71"}]: dispatch 2024-03-11T21:00:51.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:51 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-71"}]: dispatch 2024-03-11T21:00:51.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:51.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:51 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:00:51.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-71"}]: dispatch 2024-03-11T21:00:51.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:51 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-71"}]: dispatch 2024-03-11T21:00:52.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:52 smithi129 ceph-mon[42066]: pgmap v616: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:00:52.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:52 smithi129 ceph-mon[42066]: osdmap e421: 8 total, 8 up, 8 in 2024-03-11T21:00:52.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:52 smithi129 ceph-mon[30184]: pgmap v616: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:00:52.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:52 smithi129 ceph-mon[30184]: osdmap e421: 8 total, 8 up, 8 in 2024-03-11T21:00:52.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:52 smithi157 ceph-mon[33359]: pgmap v616: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:00:52.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:52 smithi157 ceph-mon[33359]: osdmap e421: 8 total, 8 up, 8 in 2024-03-11T21:00:53.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:53 smithi129 ceph-mon[30184]: osdmap e422: 8 total, 8 up, 8 in 2024-03-11T21:00:53.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:53 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:53.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:53 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:53.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:53 smithi129 ceph-mon[42066]: osdmap e422: 8 total, 8 up, 8 in 2024-03-11T21:00:53.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:53 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:53.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:53 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:53.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:53 smithi157 ceph-mon[33359]: osdmap e422: 8 total, 8 up, 8 in 2024-03-11T21:00:53.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:53 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:53.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:53 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:00:53.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:00:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:00:54.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:54 smithi129 ceph-mon[30184]: pgmap v619: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:54.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:54 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:00:54.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:00:54.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:54 smithi129 ceph-mon[30184]: osdmap e423: 8 total, 8 up, 8 in 2024-03-11T21:00:54.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:00:54.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:54 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:54.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:54 smithi129 ceph-mon[42066]: pgmap v619: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:54.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:54 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:00:54.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:00:54.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:54 smithi129 ceph-mon[42066]: osdmap e423: 8 total, 8 up, 8 in 2024-03-11T21:00:54.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:00:54.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:54 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:54.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:54 smithi157 ceph-mon[33359]: pgmap v619: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:54.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:54 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:00:54.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:00:54.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:54 smithi157 ceph-mon[33359]: osdmap e423: 8 total, 8 up, 8 in 2024-03-11T21:00:54.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:00:54.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:54 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:00:55.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:55 smithi129 ceph-mon[30184]: osdmap e424: 8 total, 8 up, 8 in 2024-03-11T21:00:55.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:55 smithi129 ceph-mon[42066]: osdmap e424: 8 total, 8 up, 8 in 2024-03-11T21:00:55.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:55 smithi157 ceph-mon[33359]: osdmap e424: 8 total, 8 up, 8 in 2024-03-11T21:00:56.380 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:56 smithi129 ceph-mon[30184]: pgmap v622: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:56.380 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:56 smithi129 ceph-mon[30184]: osdmap e425: 8 total, 8 up, 8 in 2024-03-11T21:00:56.380 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:00:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:00:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:00:56.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:56 smithi129 ceph-mon[42066]: pgmap v622: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:56.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:56 smithi129 ceph-mon[42066]: osdmap e425: 8 total, 8 up, 8 in 2024-03-11T21:00:56.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:56 smithi157 ceph-mon[33359]: pgmap v622: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:00:56.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:56 smithi157 ceph-mon[33359]: osdmap e425: 8 total, 8 up, 8 in 2024-03-11T21:00:58.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:00:58 smithi129 ceph-mon[30184]: pgmap v624: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T21:00:58.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:00:58 smithi129 ceph-mon[42066]: pgmap v624: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T21:00:58.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:00:58 smithi157 ceph-mon[33359]: pgmap v624: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-11T21:01:00.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:00 smithi129 ceph-mon[30184]: pgmap v625: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-11T21:01:00.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:00 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:01:00.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:00 smithi129 ceph-mon[42066]: pgmap v625: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-11T21:01:00.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:00 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:01:00.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:00 smithi157 ceph-mon[33359]: pgmap v625: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-11T21:01:00.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:00 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:01:02.215 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:02 smithi129 ceph-mon[30184]: pgmap v626: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 917 B/s wr, 3 op/s 2024-03-11T21:01:02.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:02 smithi129 ceph-mon[42066]: pgmap v626: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 917 B/s wr, 3 op/s 2024-03-11T21:01:02.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:02 smithi157 ceph-mon[33359]: pgmap v626: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 917 B/s wr, 3 op/s 2024-03-11T21:01:03.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:01:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:01:04.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:04 smithi129 ceph-mon[42066]: pgmap v627: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 814 B/s wr, 3 op/s 2024-03-11T21:01:04.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:04.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:04 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:04.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:04 smithi129 ceph-mon[30184]: pgmap v627: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 814 B/s wr, 3 op/s 2024-03-11T21:01:04.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:04.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:04 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:04.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:04 smithi157 ceph-mon[33359]: pgmap v627: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 814 B/s wr, 3 op/s 2024-03-11T21:01:04.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:04.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:04 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:06.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:01:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:01:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:01:06.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:06 smithi129 ceph-mon[30184]: pgmap v628: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 716 B/s wr, 3 op/s 2024-03-11T21:01:06.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:06.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:06 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:06.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-73"}]: dispatch 2024-03-11T21:01:06.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:06 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-73"}]: dispatch 2024-03-11T21:01:06.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:06 smithi129 ceph-mon[42066]: pgmap v628: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 716 B/s wr, 3 op/s 2024-03-11T21:01:06.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:06.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:06 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:06.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-73"}]: dispatch 2024-03-11T21:01:06.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:06 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-73"}]: dispatch 2024-03-11T21:01:06.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:06 smithi157 ceph-mon[33359]: pgmap v628: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 716 B/s wr, 3 op/s 2024-03-11T21:01:06.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:06.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:06 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:06.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-73"}]: dispatch 2024-03-11T21:01:06.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:06 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-73"}]: dispatch 2024-03-11T21:01:07.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:07 smithi129 ceph-mon[30184]: osdmap e426: 8 total, 8 up, 8 in 2024-03-11T21:01:07.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:07 smithi129 ceph-mon[42066]: osdmap e426: 8 total, 8 up, 8 in 2024-03-11T21:01:07.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:07 smithi157 ceph-mon[33359]: osdmap e426: 8 total, 8 up, 8 in 2024-03-11T21:01:08.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:08 smithi129 ceph-mon[30184]: pgmap v630: 260 pgs: 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:01:08.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:08 smithi129 ceph-mon[30184]: osdmap e427: 8 total, 8 up, 8 in 2024-03-11T21:01:08.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:08.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:08.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:08 smithi129 ceph-mon[42066]: pgmap v630: 260 pgs: 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:01:08.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:08 smithi129 ceph-mon[42066]: osdmap e427: 8 total, 8 up, 8 in 2024-03-11T21:01:08.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:08.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:08.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:08 smithi157 ceph-mon[33359]: pgmap v630: 260 pgs: 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:01:08.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:08 smithi157 ceph-mon[33359]: osdmap e427: 8 total, 8 up, 8 in 2024-03-11T21:01:08.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:08.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:09.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:01:09.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:09 smithi129 ceph-mon[30184]: osdmap e428: 8 total, 8 up, 8 in 2024-03-11T21:01:09.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:01:09.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:01:09.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:09 smithi129 ceph-mon[42066]: osdmap e428: 8 total, 8 up, 8 in 2024-03-11T21:01:09.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:01:09.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:01:09.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:09 smithi157 ceph-mon[33359]: osdmap e428: 8 total, 8 up, 8 in 2024-03-11T21:01:09.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:01:10.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:10 smithi129 ceph-mon[30184]: pgmap v633: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:01:10.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:10 smithi129 ceph-mon[30184]: osdmap e429: 8 total, 8 up, 8 in 2024-03-11T21:01:10.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:10 smithi129 ceph-mon[42066]: pgmap v633: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:01:10.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:10 smithi129 ceph-mon[42066]: osdmap e429: 8 total, 8 up, 8 in 2024-03-11T21:01:10.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:10 smithi157 ceph-mon[33359]: pgmap v633: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:01:10.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:10 smithi157 ceph-mon[33359]: osdmap e429: 8 total, 8 up, 8 in 2024-03-11T21:01:11.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:11 smithi129 ceph-mon[30184]: osdmap e430: 8 total, 8 up, 8 in 2024-03-11T21:01:11.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:11.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:11.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-75"}]: dispatch 2024-03-11T21:01:11.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-75"}]: dispatch 2024-03-11T21:01:11.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:11 smithi129 ceph-mon[42066]: osdmap e430: 8 total, 8 up, 8 in 2024-03-11T21:01:11.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:11.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:11.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-75"}]: dispatch 2024-03-11T21:01:11.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-75"}]: dispatch 2024-03-11T21:01:11.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:11 smithi157 ceph-mon[33359]: osdmap e430: 8 total, 8 up, 8 in 2024-03-11T21:01:11.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:11.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:11.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-75"}]: dispatch 2024-03-11T21:01:11.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-75"}]: dispatch 2024-03-11T21:01:12.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:12 smithi129 ceph-mon[30184]: pgmap v636: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:01:12.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:12 smithi129 ceph-mon[30184]: osdmap e431: 8 total, 8 up, 8 in 2024-03-11T21:01:12.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:12 smithi129 ceph-mon[42066]: pgmap v636: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:01:12.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:12 smithi129 ceph-mon[42066]: osdmap e431: 8 total, 8 up, 8 in 2024-03-11T21:01:12.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:12 smithi157 ceph-mon[33359]: pgmap v636: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:01:12.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:12 smithi157 ceph-mon[33359]: osdmap e431: 8 total, 8 up, 8 in 2024-03-11T21:01:13.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:13 smithi129 ceph-mon[30184]: osdmap e432: 8 total, 8 up, 8 in 2024-03-11T21:01:13.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:13.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:13.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:13 smithi129 ceph-mon[42066]: osdmap e432: 8 total, 8 up, 8 in 2024-03-11T21:01:13.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:13.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:13.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:13 smithi157 ceph-mon[33359]: osdmap e432: 8 total, 8 up, 8 in 2024-03-11T21:01:13.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:13.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:13.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:01:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:01:14.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:14 smithi129 ceph-mon[30184]: pgmap v639: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T21:01:14.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:14 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:01:14.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:01:14.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:14 smithi129 ceph-mon[30184]: osdmap e433: 8 total, 8 up, 8 in 2024-03-11T21:01:14.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:01:14.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:14 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:14.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:14 smithi129 ceph-mon[42066]: pgmap v639: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T21:01:14.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:14 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:01:14.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:01:14.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:14 smithi129 ceph-mon[42066]: osdmap e433: 8 total, 8 up, 8 in 2024-03-11T21:01:14.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:01:14.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:14 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:14.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:14 smithi157 ceph-mon[33359]: pgmap v639: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T21:01:14.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:14 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:01:14.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:01:14.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:14 smithi157 ceph-mon[33359]: osdmap e433: 8 total, 8 up, 8 in 2024-03-11T21:01:14.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:01:14.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:14 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:15.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:15 smithi129 ceph-mon[42066]: osdmap e434: 8 total, 8 up, 8 in 2024-03-11T21:01:15.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:15.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:15.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-77"}]: dispatch 2024-03-11T21:01:15.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-77"}]: dispatch 2024-03-11T21:01:15.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:15 smithi129 ceph-mon[30184]: osdmap e434: 8 total, 8 up, 8 in 2024-03-11T21:01:15.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:15.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:15.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-77"}]: dispatch 2024-03-11T21:01:15.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-77"}]: dispatch 2024-03-11T21:01:15.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:15 smithi157 ceph-mon[33359]: osdmap e434: 8 total, 8 up, 8 in 2024-03-11T21:01:15.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:15.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:15.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-77"}]: dispatch 2024-03-11T21:01:15.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-77"}]: dispatch 2024-03-11T21:01:16.480 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:01:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:01:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:01:16.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:16 smithi129 ceph-mon[30184]: pgmap v642: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T21:01:16.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:16 smithi129 ceph-mon[30184]: osdmap e435: 8 total, 8 up, 8 in 2024-03-11T21:01:16.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:16 smithi129 ceph-mon[42066]: pgmap v642: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T21:01:16.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:16 smithi129 ceph-mon[42066]: osdmap e435: 8 total, 8 up, 8 in 2024-03-11T21:01:16.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:16 smithi157 ceph-mon[33359]: pgmap v642: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T21:01:16.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:16 smithi157 ceph-mon[33359]: osdmap e435: 8 total, 8 up, 8 in 2024-03-11T21:01:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:17 smithi129 ceph-mon[30184]: osdmap e436: 8 total, 8 up, 8 in 2024-03-11T21:01:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:17.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:17 smithi129 ceph-mon[42066]: osdmap e436: 8 total, 8 up, 8 in 2024-03-11T21:01:17.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:17.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:17 smithi157 ceph-mon[33359]: osdmap e436: 8 total, 8 up, 8 in 2024-03-11T21:01:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:17.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:18.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:18 smithi157 ceph-mon[33359]: pgmap v645: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:01:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:01:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:18 smithi157 ceph-mon[33359]: osdmap e437: 8 total, 8 up, 8 in 2024-03-11T21:01:18.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:01:18.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:18 smithi129 ceph-mon[30184]: pgmap v645: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:01:18.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:01:18.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:18 smithi129 ceph-mon[30184]: osdmap e437: 8 total, 8 up, 8 in 2024-03-11T21:01:18.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:01:18.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:18 smithi129 ceph-mon[42066]: pgmap v645: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:01:18.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:01:18.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:18 smithi129 ceph-mon[42066]: osdmap e437: 8 total, 8 up, 8 in 2024-03-11T21:01:18.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:01:19.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:19 smithi129 ceph-mon[30184]: osdmap e438: 8 total, 8 up, 8 in 2024-03-11T21:01:19.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:19.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:19 smithi129 ceph-mon[42066]: osdmap e438: 8 total, 8 up, 8 in 2024-03-11T21:01:19.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:19.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:19 smithi157 ceph-mon[33359]: osdmap e438: 8 total, 8 up, 8 in 2024-03-11T21:01:19.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:20.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:20 smithi129 ceph-mon[30184]: pgmap v648: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:01:20.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:20 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:01:20.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:20 smithi129 ceph-mon[30184]: osdmap e439: 8 total, 8 up, 8 in 2024-03-11T21:01:20.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "304.18"}]: dispatch 2024-03-11T21:01:20.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:20 smithi129 ceph-mon[30184]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "304.18"}]: dispatch 2024-03-11T21:01:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:20 smithi129 ceph-mon[42066]: pgmap v648: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:01:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:20 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:01:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:20 smithi129 ceph-mon[42066]: osdmap e439: 8 total, 8 up, 8 in 2024-03-11T21:01:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "304.18"}]: dispatch 2024-03-11T21:01:20.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:20 smithi129 ceph-mon[42066]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "304.18"}]: dispatch 2024-03-11T21:01:20.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:20 smithi157 ceph-mon[33359]: pgmap v648: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:01:20.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:20 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:01:20.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:20 smithi157 ceph-mon[33359]: osdmap e439: 8 total, 8 up, 8 in 2024-03-11T21:01:20.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "304.18"}]: dispatch 2024-03-11T21:01:20.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:20 smithi157 ceph-mon[33359]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "304.18"}]: dispatch 2024-03-11T21:01:21.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:21 smithi129 ceph-mon[30184]: 304.18 deep-scrub starts 2024-03-11T21:01:21.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:21 smithi129 ceph-mon[30184]: 304.18 deep-scrub ok 2024-03-11T21:01:21.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:21 smithi129 ceph-mon[42066]: 304.18 deep-scrub starts 2024-03-11T21:01:21.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:21 smithi129 ceph-mon[42066]: 304.18 deep-scrub ok 2024-03-11T21:01:21.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:21 smithi157 ceph-mon[33359]: 304.18 deep-scrub starts 2024-03-11T21:01:21.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:21 smithi157 ceph-mon[33359]: 304.18 deep-scrub ok 2024-03-11T21:01:22.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:22 smithi129 ceph-mon[30184]: pgmap v650: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 436 B/s wr, 2 op/s 2024-03-11T21:01:22.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:22 smithi129 ceph-mon[42066]: pgmap v650: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 436 B/s wr, 2 op/s 2024-03-11T21:01:22.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:22 smithi157 ceph-mon[33359]: pgmap v650: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 436 B/s wr, 2 op/s 2024-03-11T21:01:23.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:01:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:01:24.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:24 smithi129 ceph-mon[42066]: pgmap v651: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 852 B/s wr, 121 op/s 2024-03-11T21:01:24.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:24 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:24.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:24 smithi129 ceph-mon[30184]: pgmap v651: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 852 B/s wr, 121 op/s 2024-03-11T21:01:24.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:24 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:24.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:24 smithi157 ceph-mon[33359]: pgmap v651: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 852 B/s wr, 121 op/s 2024-03-11T21:01:24.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:24 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:26.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:26 smithi129 ceph-mon[30184]: pgmap v652: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 665 B/s wr, 94 op/s 2024-03-11T21:01:26.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:26 smithi129 ceph-mon[42066]: pgmap v652: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 665 B/s wr, 94 op/s 2024-03-11T21:01:26.480 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:01:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:01:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:01:26.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:26 smithi157 ceph-mon[33359]: pgmap v652: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 665 B/s wr, 94 op/s 2024-03-11T21:01:28.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:28 smithi129 ceph-mon[30184]: pgmap v653: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 590 B/s wr, 84 op/s 2024-03-11T21:01:28.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:28 smithi129 ceph-mon[42066]: pgmap v653: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 590 B/s wr, 84 op/s 2024-03-11T21:01:28.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:28 smithi157 ceph-mon[33359]: pgmap v653: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 590 B/s wr, 84 op/s 2024-03-11T21:01:30.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:30 smithi129 ceph-mon[30184]: pgmap v654: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 511 B/s wr, 73 op/s 2024-03-11T21:01:30.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:30 smithi129 ceph-mon[42066]: pgmap v654: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 511 B/s wr, 73 op/s 2024-03-11T21:01:30.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:30 smithi157 ceph-mon[33359]: pgmap v654: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 511 B/s wr, 73 op/s 2024-03-11T21:01:32.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:32 smithi129 ceph-mon[30184]: pgmap v655: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 438 B/s wr, 63 op/s 2024-03-11T21:01:32.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:32 smithi129 ceph-mon[42066]: pgmap v655: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 438 B/s wr, 63 op/s 2024-03-11T21:01:32.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:32 smithi157 ceph-mon[33359]: pgmap v655: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 438 B/s wr, 63 op/s 2024-03-11T21:01:33.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:01:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:01:34.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:34 smithi129 ceph-mon[30184]: pgmap v656: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 426 B/s wr, 61 op/s 2024-03-11T21:01:34.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:34 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:34.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:34 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:34.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:34 smithi129 ceph-mon[42066]: pgmap v656: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 426 B/s wr, 61 op/s 2024-03-11T21:01:34.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:34 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:34.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:34 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:34.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:34 smithi157 ceph-mon[33359]: pgmap v656: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 426 B/s wr, 61 op/s 2024-03-11T21:01:34.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:34 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:34.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:34 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:36.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:36 smithi129 ceph-mon[30184]: pgmap v657: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:01:36.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:36 smithi129 ceph-mon[42066]: pgmap v657: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:01:36.480 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:01:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:01:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:01:36.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:36 smithi157 ceph-mon[33359]: pgmap v657: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:01:38.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:38 smithi129 ceph-mon[30184]: pgmap v658: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:01:38.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:38 smithi129 ceph-mon[42066]: pgmap v658: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:01:38.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:38 smithi157 ceph-mon[33359]: pgmap v658: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:01:39.729 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 21:01:39 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-2[58560]: 2024-03-11T21:01:39.274+0000 7f3b4c793700 -1 snap_mapper.add_oid found existing snaps mapped on 304:1aec152a:test-rados-api-smithi129-82818-80::foo:2, removing 2024-03-11T21:01:39.736 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:01:39 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:01:39.275+0000 7f86cfdb1700 -1 snap_mapper.add_oid found existing snaps mapped on 304:1aec152a:test-rados-api-smithi129-82818-80::foo:2, removing 2024-03-11T21:01:39.737 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:01:39 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:01:39.275+0000 7f257fc9d700 -1 snap_mapper.add_oid found existing snaps mapped on 304:1aec152a:test-rados-api-smithi129-82818-80::foo:2, removing 2024-03-11T21:01:40.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:40 smithi129 ceph-mon[30184]: pgmap v659: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:01:40.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:40.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:40.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-79"}]: dispatch 2024-03-11T21:01:40.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:40 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-79"}]: dispatch 2024-03-11T21:01:40.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:40 smithi129 ceph-mon[42066]: pgmap v659: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:01:40.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:40.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:40.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-79"}]: dispatch 2024-03-11T21:01:40.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:40 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-79"}]: dispatch 2024-03-11T21:01:40.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:40 smithi157 ceph-mon[33359]: pgmap v659: 292 pgs: 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:01:40.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:40.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:40.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-79"}]: dispatch 2024-03-11T21:01:40.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:40 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-79"}]: dispatch 2024-03-11T21:01:41.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:41 smithi129 ceph-mon[30184]: osdmap e440: 8 total, 8 up, 8 in 2024-03-11T21:01:41.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:41 smithi129 ceph-mon[42066]: osdmap e440: 8 total, 8 up, 8 in 2024-03-11T21:01:41.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:41 smithi157 ceph-mon[33359]: osdmap e440: 8 total, 8 up, 8 in 2024-03-11T21:01:42.638 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:42 smithi157 ceph-mon[33359]: pgmap v661: 260 pgs: 260 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:01:42.638 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:42 smithi157 ceph-mon[33359]: osdmap e441: 8 total, 8 up, 8 in 2024-03-11T21:01:42.638 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:42.638 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:42 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:42.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:42 smithi129 ceph-mon[30184]: pgmap v661: 260 pgs: 260 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:01:42.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:42 smithi129 ceph-mon[30184]: osdmap e441: 8 total, 8 up, 8 in 2024-03-11T21:01:42.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:42.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:42 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:42.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:42 smithi129 ceph-mon[42066]: pgmap v661: 260 pgs: 260 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:01:42.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:42 smithi129 ceph-mon[42066]: osdmap e441: 8 total, 8 up, 8 in 2024-03-11T21:01:42.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:42.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:42 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:43.676 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:01:43.676 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:43 smithi157 ceph-mon[33359]: osdmap e442: 8 total, 8 up, 8 in 2024-03-11T21:01:43.676 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:01:43.676 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:01:43.676 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:43 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:01:43.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:01:43.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:43 smithi129 ceph-mon[30184]: osdmap e442: 8 total, 8 up, 8 in 2024-03-11T21:01:43.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:01:43.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:01:43.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:43 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:01:43.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:01:43.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:43 smithi129 ceph-mon[42066]: osdmap e442: 8 total, 8 up, 8 in 2024-03-11T21:01:43.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:01:43.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:01:43.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:43 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:01:43.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:01:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:01:44.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:44 smithi129 ceph-mon[30184]: pgmap v664: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:01:44.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:44 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:01:44.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:44 smithi129 ceph-mon[30184]: osdmap e443: 8 total, 8 up, 8 in 2024-03-11T21:01:44.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:44.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:44 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:44.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:44 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:44.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:44 smithi129 ceph-mon[42066]: pgmap v664: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:01:44.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:44 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:01:44.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:44 smithi129 ceph-mon[42066]: osdmap e443: 8 total, 8 up, 8 in 2024-03-11T21:01:44.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:44.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:44 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:44.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:44 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:44.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:44 smithi157 ceph-mon[33359]: pgmap v664: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:01:44.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:44 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:01:44.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:44 smithi157 ceph-mon[33359]: osdmap e443: 8 total, 8 up, 8 in 2024-03-11T21:01:44.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:44.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:44 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:44.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:44 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:45.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:45 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T21:01:45.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:45 smithi129 ceph-mon[30184]: osdmap e444: 8 total, 8 up, 8 in 2024-03-11T21:01:45.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:01:45.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:45 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:01:45.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:45 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:01:45.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:45 smithi129 ceph-mon[30184]: osdmap e445: 8 total, 8 up, 8 in 2024-03-11T21:01:45.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:45 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T21:01:45.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:45 smithi129 ceph-mon[42066]: osdmap e444: 8 total, 8 up, 8 in 2024-03-11T21:01:45.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:01:45.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:45 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:01:45.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:45 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:01:45.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:45 smithi129 ceph-mon[42066]: osdmap e445: 8 total, 8 up, 8 in 2024-03-11T21:01:45.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:45 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T21:01:45.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:45 smithi157 ceph-mon[33359]: osdmap e444: 8 total, 8 up, 8 in 2024-03-11T21:01:45.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:01:45.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:45 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:01:45.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:45 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:01:45.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:45 smithi157 ceph-mon[33359]: osdmap e445: 8 total, 8 up, 8 in 2024-03-11T21:01:46.170 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4c050 2024-03-11T21:01:46.171 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4c180 2024-03-11T21:01:46.171 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4ccd0 2024-03-11T21:01:46.171 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4d180 2024-03-11T21:01:46.171 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4d290 2024-03-11T21:01:46.171 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4d870 2024-03-11T21:01:46.171 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4cde0 2024-03-11T21:01:46.171 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4cf30 2024-03-11T21:01:46.171 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4d3c0 2024-03-11T21:01:46.171 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4d4f0 2024-03-11T21:01:46.171 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4d620 2024-03-11T21:01:46.172 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4d750 2024-03-11T21:01:46.172 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4de70 2024-03-11T21:01:46.172 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4d9c0 2024-03-11T21:01:46.172 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4db10 2024-03-11T21:01:46.172 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4dc40 2024-03-11T21:01:46.172 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4dfa0 2024-03-11T21:01:46.172 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4e470 2024-03-11T21:01:46.172 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4e5a0 2024-03-11T21:01:46.172 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4c6b0 2024-03-11T21:01:46.172 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4c7e0 2024-03-11T21:01:46.172 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4c910 2024-03-11T21:01:46.173 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4e6d0 2024-03-11T21:01:46.173 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4eba0 2024-03-11T21:01:46.173 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4ecf0 2024-03-11T21:01:46.173 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4e0d0 2024-03-11T21:01:46.173 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4e200 2024-03-11T21:01:46.173 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4e330 2024-03-11T21:01:46.173 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4ee20 2024-03-11T21:01:46.173 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4f2f0 2024-03-11T21:01:46.173 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4f440 2024-03-11T21:01:46.173 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4dd50 2024-03-11T21:01:46.173 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4f910 2024-03-11T21:01:46.173 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4f570 2024-03-11T21:01:46.173 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4f6c0 2024-03-11T21:01:46.173 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4ef30 2024-03-11T21:01:46.174 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4f080 2024-03-11T21:01:46.174 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4f1b0 2024-03-11T21:01:46.174 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4fed0 2024-03-11T21:01:46.174 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4fa20 2024-03-11T21:01:46.174 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4fb70 2024-03-11T21:01:46.174 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4fca0 2024-03-11T21:01:46.174 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a50000 2024-03-11T21:01:46.174 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a504d0 2024-03-11T21:01:46.174 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a50600 2024-03-11T21:01:46.174 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4fdb0 2024-03-11T21:01:46.174 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a50730 2024-03-11T21:01:46.174 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a50880 2024-03-11T21:01:46.174 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a50110 2024-03-11T21:01:46.174 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a50260 2024-03-11T21:01:46.175 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a50390 2024-03-11T21:01:46.175 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a509d0 2024-03-11T21:01:46.175 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a50ea0 2024-03-11T21:01:46.175 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a50fd0 2024-03-11T21:01:46.175 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4e800 2024-03-11T21:01:46.175 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4e950 2024-03-11T21:01:46.175 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a4ea80 2024-03-11T21:01:46.175 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a510e0 2024-03-11T21:01:46.175 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a515b0 2024-03-11T21:01:46.175 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a516e0 2024-03-11T21:01:46.175 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a50b00 2024-03-11T21:01:46.175 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a50c50 2024-03-11T21:01:46.175 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a50d80 2024-03-11T21:01:46.175 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a517f0 2024-03-11T21:01:46.176 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a51960 2024-03-11T21:01:46.176 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a51f60 2024-03-11T21:01:46.176 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a51ab0 2024-03-11T21:01:46.176 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a51be0 2024-03-11T21:01:46.176 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a51d10 2024-03-11T21:01:46.176 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a51e40 2024-03-11T21:01:46.176 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a52540 2024-03-11T21:01:46.176 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a52090 2024-03-11T21:01:46.176 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a521e0 2024-03-11T21:01:46.176 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a52310 2024-03-11T21:01:46.176 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a52670 2024-03-11T21:01:46.176 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a52b90 2024-03-11T21:01:46.176 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a52cc0 2024-03-11T21:01:46.177 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a511f0 2024-03-11T21:01:46.177 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a51320 2024-03-11T21:01:46.177 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a51450 2024-03-11T21:01:46.177 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a52df0 2024-03-11T21:01:46.177 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a532c0 2024-03-11T21:01:46.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:01:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:01:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:01:46.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:46 smithi129 ceph-mon[30184]: pgmap v667: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:01:46.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:46.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:46 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:46.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:46 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:01:46.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:46 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:01:46.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:46 smithi129 ceph-mon[30184]: osdmap e446: 8 total, 8 up, 8 in 2024-03-11T21:01:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:46 smithi129 ceph-mon[42066]: pgmap v667: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:01:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:46 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:46 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:46 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:01:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:46 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:01:46.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:46 smithi129 ceph-mon[42066]: osdmap e446: 8 total, 8 up, 8 in 2024-03-11T21:01:46.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:46 smithi157 ceph-mon[33359]: pgmap v667: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:01:46.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:46 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:46.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:46 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:46.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:46 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:01:46.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:46 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:01:46.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:46 smithi157 ceph-mon[33359]: osdmap e446: 8 total, 8 up, 8 in 2024-03-11T21:01:47.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-11T21:01:47.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:47 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-11T21:01:47.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:47 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:01:47.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:47 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:01:47.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:47 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:01:47.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-11T21:01:47.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:47 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-11T21:01:47.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:47 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:01:47.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:47 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:01:47.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:47 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:01:47.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-11T21:01:47.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:47 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-11T21:01:47.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:47 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:01:47.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:47 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:01:47.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:47 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:01:48.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:48 smithi129 ceph-mon[42066]: pgmap v670: 292 pgs: 292 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-11T21:01:48.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:48 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-11T21:01:48.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:48 smithi129 ceph-mon[42066]: osdmap e447: 8 total, 8 up, 8 in 2024-03-11T21:01:48.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-11T21:01:48.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:48 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-11T21:01:48.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:48 smithi129 ceph-mon[30184]: pgmap v670: 292 pgs: 292 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-11T21:01:48.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:48 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-11T21:01:48.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:48 smithi129 ceph-mon[30184]: osdmap e447: 8 total, 8 up, 8 in 2024-03-11T21:01:48.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-11T21:01:48.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:48 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-11T21:01:48.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:48 smithi157 ceph-mon[33359]: pgmap v670: 292 pgs: 292 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-11T21:01:48.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:48 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-11T21:01:48.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:48 smithi157 ceph-mon[33359]: osdmap e447: 8 total, 8 up, 8 in 2024-03-11T21:01:48.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-11T21:01:48.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:48 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-11T21:01:49.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-11T21:01:49.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:49 smithi129 ceph-mon[30184]: osdmap e448: 8 total, 8 up, 8 in 2024-03-11T21:01:49.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:49.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:49.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:49 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:49.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-11T21:01:49.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:49 smithi129 ceph-mon[42066]: osdmap e448: 8 total, 8 up, 8 in 2024-03-11T21:01:49.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:49.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:49.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:49 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:49.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-11T21:01:49.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:49 smithi157 ceph-mon[33359]: osdmap e448: 8 total, 8 up, 8 in 2024-03-11T21:01:49.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:49.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:49.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:49 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:01:50.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:50 smithi129 ceph-mon[30184]: pgmap v673: 292 pgs: 292 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-11T21:01:50.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:01:50.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:50 smithi129 ceph-mon[30184]: osdmap e449: 8 total, 8 up, 8 in 2024-03-11T21:01:50.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:50.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:50.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-81"}]: dispatch 2024-03-11T21:01:50.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-81"}]: dispatch 2024-03-11T21:01:50.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:50 smithi129 ceph-mon[42066]: pgmap v673: 292 pgs: 292 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-11T21:01:50.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:01:50.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:50 smithi129 ceph-mon[42066]: osdmap e449: 8 total, 8 up, 8 in 2024-03-11T21:01:50.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:50.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:50.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-81"}]: dispatch 2024-03-11T21:01:50.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-81"}]: dispatch 2024-03-11T21:01:50.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:50 smithi157 ceph-mon[33359]: pgmap v673: 292 pgs: 292 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 5.2 KiB/s wr, 8 op/s 2024-03-11T21:01:50.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:01:50.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:50 smithi157 ceph-mon[33359]: osdmap e449: 8 total, 8 up, 8 in 2024-03-11T21:01:50.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:50.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:50.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-81"}]: dispatch 2024-03-11T21:01:50.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-81"}]: dispatch 2024-03-11T21:01:51.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:51 smithi129 ceph-mon[30184]: osdmap e450: 8 total, 8 up, 8 in 2024-03-11T21:01:51.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:51 smithi129 ceph-mon[42066]: osdmap e450: 8 total, 8 up, 8 in 2024-03-11T21:01:51.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:51 smithi157 ceph-mon[33359]: osdmap e450: 8 total, 8 up, 8 in 2024-03-11T21:01:52.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:52 smithi129 ceph-mon[30184]: pgmap v676: 260 pgs: 260 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:01:52.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:52 smithi129 ceph-mon[30184]: osdmap e451: 8 total, 8 up, 8 in 2024-03-11T21:01:52.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:52.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:52.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:52 smithi129 ceph-mon[42066]: pgmap v676: 260 pgs: 260 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:01:52.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:52 smithi129 ceph-mon[42066]: osdmap e451: 8 total, 8 up, 8 in 2024-03-11T21:01:52.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:52.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:52.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:52 smithi157 ceph-mon[33359]: pgmap v676: 260 pgs: 260 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:01:52.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:52 smithi157 ceph-mon[33359]: osdmap e451: 8 total, 8 up, 8 in 2024-03-11T21:01:52.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:52.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:01:53.687 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:53 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:01:53.687 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:53 smithi157 ceph-mon[33359]: osdmap e452: 8 total, 8 up, 8 in 2024-03-11T21:01:53.687 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:53 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:01:53.687 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:53 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:01:53.687 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:53 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:01:53.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:53 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:01:53.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:53 smithi129 ceph-mon[30184]: osdmap e452: 8 total, 8 up, 8 in 2024-03-11T21:01:53.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:53 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:01:53.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:53 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:01:53.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:53 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:01:53.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:53 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:01:53.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:53 smithi129 ceph-mon[42066]: osdmap e452: 8 total, 8 up, 8 in 2024-03-11T21:01:53.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:53 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:01:53.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:53 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:01:53.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:53 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:01:53.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:01:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:01:54.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:54 smithi129 ceph-mon[30184]: pgmap v679: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-03-11T21:01:54.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:01:54.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:54 smithi129 ceph-mon[30184]: osdmap e453: 8 total, 8 up, 8 in 2024-03-11T21:01:54.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:54.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:54.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:54 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:54.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:54 smithi129 ceph-mon[42066]: pgmap v679: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-03-11T21:01:54.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:01:54.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:54 smithi129 ceph-mon[42066]: osdmap e453: 8 total, 8 up, 8 in 2024-03-11T21:01:54.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:54.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:54.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:54 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:54.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:54 smithi157 ceph-mon[33359]: pgmap v679: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 32 op/s 2024-03-11T21:01:54.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:01:54.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:54 smithi157 ceph-mon[33359]: osdmap e453: 8 total, 8 up, 8 in 2024-03-11T21:01:54.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:54.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:54.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:54 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:01:55.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T21:01:55.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:01:55.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:55 smithi129 ceph-mon[30184]: osdmap e454: 8 total, 8 up, 8 in 2024-03-11T21:01:55.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:01:55.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:01:55.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:55 smithi129 ceph-mon[30184]: osdmap e455: 8 total, 8 up, 8 in 2024-03-11T21:01:55.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:55.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:55.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T21:01:55.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:01:55.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:55 smithi129 ceph-mon[42066]: osdmap e454: 8 total, 8 up, 8 in 2024-03-11T21:01:55.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:01:55.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:01:55.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:55 smithi129 ceph-mon[42066]: osdmap e455: 8 total, 8 up, 8 in 2024-03-11T21:01:55.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:55.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:55.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T21:01:55.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:01:55.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:55 smithi157 ceph-mon[33359]: osdmap e454: 8 total, 8 up, 8 in 2024-03-11T21:01:55.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:01:55.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:01:55.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:55 smithi157 ceph-mon[33359]: osdmap e455: 8 total, 8 up, 8 in 2024-03-11T21:01:55.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:55.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:01:56.401 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:01:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:01:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:01:56.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:56 smithi129 ceph-mon[30184]: pgmap v682: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2024-03-11T21:01:56.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:56 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:01:56.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:56 smithi129 ceph-mon[30184]: osdmap e456: 8 total, 8 up, 8 in 2024-03-11T21:01:56.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:56 smithi129 ceph-mon[42066]: pgmap v682: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2024-03-11T21:01:56.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:56 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:01:56.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:56 smithi129 ceph-mon[42066]: osdmap e456: 8 total, 8 up, 8 in 2024-03-11T21:01:56.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:56 smithi157 ceph-mon[33359]: pgmap v682: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 9.5 KiB/s rd, 9.0 KiB/s wr, 32 op/s 2024-03-11T21:01:56.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:56 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:01:56.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:56 smithi157 ceph-mon[33359]: osdmap e456: 8 total, 8 up, 8 in 2024-03-11T21:01:58.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:58 smithi129 ceph-mon[30184]: pgmap v685: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-11T21:01:58.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:58 smithi129 ceph-mon[30184]: osdmap e457: 8 total, 8 up, 8 in 2024-03-11T21:01:58.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:58 smithi129 ceph-mon[42066]: pgmap v685: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-11T21:01:58.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:58 smithi129 ceph-mon[42066]: osdmap e457: 8 total, 8 up, 8 in 2024-03-11T21:01:58.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:58 smithi157 ceph-mon[33359]: pgmap v685: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-11T21:01:58.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:58 smithi157 ceph-mon[33359]: osdmap e457: 8 total, 8 up, 8 in 2024-03-11T21:01:59.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:59 smithi129 ceph-mon[30184]: osdmap e458: 8 total, 8 up, 8 in 2024-03-11T21:01:59.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:59.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:59 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:59.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-83"}]: dispatch 2024-03-11T21:01:59.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:01:59 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-83"}]: dispatch 2024-03-11T21:01:59.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:59 smithi129 ceph-mon[42066]: osdmap e458: 8 total, 8 up, 8 in 2024-03-11T21:01:59.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:59.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:59 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:59.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-83"}]: dispatch 2024-03-11T21:01:59.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:01:59 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-83"}]: dispatch 2024-03-11T21:01:59.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:59 smithi157 ceph-mon[33359]: osdmap e458: 8 total, 8 up, 8 in 2024-03-11T21:01:59.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:59.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:59 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:01:59.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-83"}]: dispatch 2024-03-11T21:01:59.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:01:59 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-83"}]: dispatch 2024-03-11T21:02:00.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:00 smithi129 ceph-mon[42066]: pgmap v688: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-11T21:02:00.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:00 smithi129 ceph-mon[42066]: osdmap e459: 8 total, 8 up, 8 in 2024-03-11T21:02:00.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:00 smithi129 ceph-mon[30184]: pgmap v688: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-11T21:02:00.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:00 smithi129 ceph-mon[30184]: osdmap e459: 8 total, 8 up, 8 in 2024-03-11T21:02:00.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:00 smithi157 ceph-mon[33359]: pgmap v688: 292 pgs: 292 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-11T21:02:00.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:00 smithi157 ceph-mon[33359]: osdmap e459: 8 total, 8 up, 8 in 2024-03-11T21:02:01.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:01 smithi129 ceph-mon[30184]: osdmap e460: 8 total, 8 up, 8 in 2024-03-11T21:02:01.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:01.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:01 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:01.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:01 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:01.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:01 smithi129 ceph-mon[42066]: osdmap e460: 8 total, 8 up, 8 in 2024-03-11T21:02:01.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:01.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:01 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:01.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:01 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:01.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:01 smithi157 ceph-mon[33359]: osdmap e460: 8 total, 8 up, 8 in 2024-03-11T21:02:01.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:01.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:01 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:01.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:01 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:02.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[30184]: pgmap v691: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:02.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[30184]: osdmap e461: 8 total, 8 up, 8 in 2024-03-11T21:02:02.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:02.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:02.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:02.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:02.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:02:02.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[30184]: osdmap e462: 8 total, 8 up, 8 in 2024-03-11T21:02:02.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:02.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:02.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[42066]: pgmap v691: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:02.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[42066]: osdmap e461: 8 total, 8 up, 8 in 2024-03-11T21:02:02.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:02.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:02.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:02.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:02.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:02:02.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[42066]: osdmap e462: 8 total, 8 up, 8 in 2024-03-11T21:02:02.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:02.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:02 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:02.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:02 smithi157 ceph-mon[33359]: pgmap v691: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:02.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:02 smithi157 ceph-mon[33359]: osdmap e461: 8 total, 8 up, 8 in 2024-03-11T21:02:02.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:02 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:02.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:02 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:02.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:02 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:02.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:02 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:02.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:02 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:02:02.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:02 smithi157 ceph-mon[33359]: osdmap e462: 8 total, 8 up, 8 in 2024-03-11T21:02:02.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:02 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:02.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:02 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:03.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:02:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:02:04.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:04 smithi129 ceph-mon[30184]: pgmap v694: 292 pgs: 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-11T21:02:04.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T21:02:04.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:04 smithi129 ceph-mon[30184]: osdmap e463: 8 total, 8 up, 8 in 2024-03-11T21:02:04.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:02:04.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:02:04.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:04.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:04 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:04 smithi129 ceph-mon[42066]: pgmap v694: 292 pgs: 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-11T21:02:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T21:02:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:04 smithi129 ceph-mon[42066]: osdmap e463: 8 total, 8 up, 8 in 2024-03-11T21:02:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:02:04.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:02:04.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:04.731 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:04 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:04.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:04 smithi157 ceph-mon[33359]: pgmap v694: 292 pgs: 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-11T21:02:04.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_tier","val": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T21:02:04.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:04 smithi157 ceph-mon[33359]: osdmap e463: 8 total, 8 up, 8 in 2024-03-11T21:02:04.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:02:04.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:02:04.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:04.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:04 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:05.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:05 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:02:05.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:05 smithi129 ceph-mon[30184]: osdmap e464: 8 total, 8 up, 8 in 2024-03-11T21:02:05.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:02:05.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:05 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:02:05.730 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:05 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:02:05.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:05 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:02:05.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:05 smithi129 ceph-mon[42066]: osdmap e464: 8 total, 8 up, 8 in 2024-03-11T21:02:05.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:02:05.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:05 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:02:05.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:05 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:02:05.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:05 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:02:05.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:05 smithi157 ceph-mon[33359]: osdmap e464: 8 total, 8 up, 8 in 2024-03-11T21:02:05.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:02:05.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:05 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:02:05.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:05 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:02:06.460 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:02:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:02:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:02:06.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:06 smithi129 ceph-mon[30184]: pgmap v697: 292 pgs: 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-11T21:02:06.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:06 smithi129 ceph-mon[30184]: osdmap e465: 8 total, 8 up, 8 in 2024-03-11T21:02:06.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:06 smithi129 ceph-mon[42066]: pgmap v697: 292 pgs: 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-11T21:02:06.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:06 smithi129 ceph-mon[42066]: osdmap e465: 8 total, 8 up, 8 in 2024-03-11T21:02:06.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:06 smithi157 ceph-mon[33359]: pgmap v697: 292 pgs: 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-03-11T21:02:06.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:06 smithi157 ceph-mon[33359]: osdmap e465: 8 total, 8 up, 8 in 2024-03-11T21:02:07.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:07 smithi129 ceph-mon[30184]: osdmap e466: 8 total, 8 up, 8 in 2024-03-11T21:02:07.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:07 smithi129 ceph-mon[42066]: osdmap e466: 8 total, 8 up, 8 in 2024-03-11T21:02:07.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:07 smithi157 ceph-mon[33359]: osdmap e466: 8 total, 8 up, 8 in 2024-03-11T21:02:08.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:08 smithi129 ceph-mon[30184]: pgmap v700: 292 pgs: 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-11T21:02:08.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:08 smithi129 ceph-mon[30184]: osdmap e467: 8 total, 8 up, 8 in 2024-03-11T21:02:08.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:08.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:08.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-85"}]: dispatch 2024-03-11T21:02:08.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-85"}]: dispatch 2024-03-11T21:02:08.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:08 smithi129 ceph-mon[42066]: pgmap v700: 292 pgs: 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-11T21:02:08.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:08 smithi129 ceph-mon[42066]: osdmap e467: 8 total, 8 up, 8 in 2024-03-11T21:02:08.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:08.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:08.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-85"}]: dispatch 2024-03-11T21:02:08.730 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-85"}]: dispatch 2024-03-11T21:02:08.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:08 smithi157 ceph-mon[33359]: pgmap v700: 292 pgs: 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-03-11T21:02:08.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:08 smithi157 ceph-mon[33359]: osdmap e467: 8 total, 8 up, 8 in 2024-03-11T21:02:08.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:08.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:08.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-85"}]: dispatch 2024-03-11T21:02:08.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-85"}]: dispatch 2024-03-11T21:02:09.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:09 smithi129 ceph-mon[30184]: osdmap e468: 8 total, 8 up, 8 in 2024-03-11T21:02:09.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:09 smithi129 ceph-mon[42066]: osdmap e468: 8 total, 8 up, 8 in 2024-03-11T21:02:09.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:09 smithi157 ceph-mon[33359]: osdmap e468: 8 total, 8 up, 8 in 2024-03-11T21:02:10.514 INFO:tasks.workunit.client.0.smithi129.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18074 ms) 2024-03-11T21:02:10.514 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-03-11T21:02:10.514 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22428 ms) 2024-03-11T21:02:10.514 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-03-11T21:02:10.514 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3012 ms) 2024-03-11T21:02:10.514 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-03-11T21:02:10.514 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:02:10.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3026 ms) 2024-03-11T21:02:10.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-03-11T21:02:10.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3015 ms) 2024-03-11T21:02:10.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-03-11T21:02:10.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:02:10.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3023 ms) 2024-03-11T21:02:10.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-03-11T21:02:10.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:02:10.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4028 ms) 2024-03-11T21:02:10.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-03-11T21:02:10.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:02:10.515 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38132 ms) 2024-03-11T21:02:10.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-03-11T21:02:10.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:02:10.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16215 ms) 2024-03-11T21:02:10.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-03-11T21:02:10.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:02:10.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4025 ms) 2024-03-11T21:02:10.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-03-11T21:02:10.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:02:10.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3021 ms) 2024-03-11T21:02:10.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-03-11T21:02:10.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:02:10.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25089 ms) 2024-03-11T21:02:10.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-03-11T21:02:10.516 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:02:10.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15051 ms) 2024-03-11T21:02:10.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-03-11T21:02:10.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:02:10.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5055 ms) 2024-03-11T21:02:10.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-03-11T21:02:10.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:02:10.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4022 ms) 2024-03-11T21:02:10.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-03-11T21:02:10.517 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:02:10.518 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: waiting for scrubs... 2024-03-11T21:02:10.518 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: done waiting 2024-03-11T21:02:10.518 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25078 ms) 2024-03-11T21:02:10.518 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-03-11T21:02:10.518 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:02:10.518 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10074 ms) 2024-03-11T21:02:10.518 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-03-11T21:02:10.518 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:02:10.518 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9073 ms) 2024-03-11T21:02:10.518 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-03-11T21:02:10.518 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:02:10.518 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9061 ms) 2024-03-11T21:02:10.518 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-03-11T21:02:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:10 smithi129 ceph-mon[30184]: pgmap v703: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:02:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:10 smithi129 ceph-mon[30184]: osdmap e469: 8 total, 8 up, 8 in 2024-03-11T21:02:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:10.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:10 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:10.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:10 smithi129 ceph-mon[42066]: pgmap v703: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:02:10.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:10 smithi129 ceph-mon[42066]: osdmap e469: 8 total, 8 up, 8 in 2024-03-11T21:02:10.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:10.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:10.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:10 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:10.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:10 smithi157 ceph-mon[33359]: pgmap v703: 260 pgs: 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:02:10.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:10 smithi157 ceph-mon[33359]: osdmap e469: 8 total, 8 up, 8 in 2024-03-11T21:02:10.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:10.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:10.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:10 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:11.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:11.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:11 smithi129 ceph-mon[42066]: osdmap e470: 8 total, 8 up, 8 in 2024-03-11T21:02:11.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:11.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:11.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:11 smithi129 ceph-mon[30184]: osdmap e470: 8 total, 8 up, 8 in 2024-03-11T21:02:11.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:11.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:11.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:11 smithi157 ceph-mon[33359]: osdmap e470: 8 total, 8 up, 8 in 2024-03-11T21:02:11.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:12.888 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:12 smithi157 ceph-mon[33359]: pgmap v706: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:12.888 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:12 smithi157 ceph-mon[33359]: osdmap e471: 8 total, 8 up, 8 in 2024-03-11T21:02:12.888 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:12.888 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:12 smithi129 ceph-mon[30184]: pgmap v706: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:12 smithi129 ceph-mon[30184]: osdmap e471: 8 total, 8 up, 8 in 2024-03-11T21:02:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:12.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:12.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:12 smithi129 ceph-mon[42066]: pgmap v706: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:12.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:12 smithi129 ceph-mon[42066]: osdmap e471: 8 total, 8 up, 8 in 2024-03-11T21:02:12.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:12.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:13.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:02:13.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:13 smithi129 ceph-mon[30184]: osdmap e472: 8 total, 8 up, 8 in 2024-03-11T21:02:13.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_tier","val": "test-rados-api-smithi129-82818-89-test-flush"}]: dispatch 2024-03-11T21:02:13.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_tier","val": "test-rados-api-smithi129-82818-89-test-flush"}]: dispatch 2024-03-11T21:02:13.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_tier","val": "test-rados-api-smithi129-82818-89-test-flush"}]': finished 2024-03-11T21:02:13.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:13 smithi129 ceph-mon[30184]: osdmap e473: 8 total, 8 up, 8 in 2024-03-11T21:02:13.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:02:13.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:02:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:02:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:13 smithi129 ceph-mon[42066]: osdmap e472: 8 total, 8 up, 8 in 2024-03-11T21:02:13.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_tier","val": "test-rados-api-smithi129-82818-89-test-flush"}]: dispatch 2024-03-11T21:02:13.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_tier","val": "test-rados-api-smithi129-82818-89-test-flush"}]: dispatch 2024-03-11T21:02:13.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_tier","val": "test-rados-api-smithi129-82818-89-test-flush"}]': finished 2024-03-11T21:02:13.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:13 smithi129 ceph-mon[42066]: osdmap e473: 8 total, 8 up, 8 in 2024-03-11T21:02:13.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:02:13.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:02:13.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:02:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:02:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:02:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:13 smithi157 ceph-mon[33359]: osdmap e472: 8 total, 8 up, 8 in 2024-03-11T21:02:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_tier","val": "test-rados-api-smithi129-82818-89-test-flush"}]: dispatch 2024-03-11T21:02:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_tier","val": "test-rados-api-smithi129-82818-89-test-flush"}]: dispatch 2024-03-11T21:02:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_tier","val": "test-rados-api-smithi129-82818-89-test-flush"}]': finished 2024-03-11T21:02:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:13 smithi157 ceph-mon[33359]: osdmap e473: 8 total, 8 up, 8 in 2024-03-11T21:02:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:02:13.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:02:14.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:14 smithi129 ceph-mon[30184]: pgmap v709: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-11T21:02:14.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:14 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:14.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:02:14.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:14 smithi129 ceph-mon[30184]: osdmap e474: 8 total, 8 up, 8 in 2024-03-11T21:02:14.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:02:14.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:02:14.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:14 smithi129 ceph-mon[42066]: pgmap v709: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-11T21:02:14.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:14 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:14.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:02:14.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:14 smithi129 ceph-mon[42066]: osdmap e474: 8 total, 8 up, 8 in 2024-03-11T21:02:14.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:02:14.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:02:14.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:14 smithi157 ceph-mon[33359]: pgmap v709: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-11T21:02:14.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:14 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:14.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:02:14.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:14 smithi157 ceph-mon[33359]: osdmap e474: 8 total, 8 up, 8 in 2024-03-11T21:02:14.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:02:14.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:02:15.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:15 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:15.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:15 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:02:15.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:15 smithi129 ceph-mon[30184]: osdmap e475: 8 total, 8 up, 8 in 2024-03-11T21:02:15.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:15 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:15.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:15 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:02:15.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:15 smithi129 ceph-mon[42066]: osdmap e475: 8 total, 8 up, 8 in 2024-03-11T21:02:15.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:15 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:15.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:15 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:02:15.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:15 smithi157 ceph-mon[33359]: osdmap e475: 8 total, 8 up, 8 in 2024-03-11T21:02:16.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:02:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:02:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:02:16.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:16 smithi129 ceph-mon[30184]: pgmap v712: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-11T21:02:16.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:16 smithi129 ceph-mon[42066]: pgmap v712: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-11T21:02:16.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:16 smithi157 ceph-mon[33359]: pgmap v712: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-11T21:02:17.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:17 smithi129 ceph-mon[30184]: osdmap e476: 8 total, 8 up, 8 in 2024-03-11T21:02:17.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:17.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:17.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-87"}]: dispatch 2024-03-11T21:02:17.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-87"}]: dispatch 2024-03-11T21:02:17.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:17 smithi129 ceph-mon[42066]: osdmap e476: 8 total, 8 up, 8 in 2024-03-11T21:02:17.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:17.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:17.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-87"}]: dispatch 2024-03-11T21:02:17.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-87"}]: dispatch 2024-03-11T21:02:17.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:17 smithi157 ceph-mon[33359]: osdmap e476: 8 total, 8 up, 8 in 2024-03-11T21:02:17.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:17.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:17.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-87"}]: dispatch 2024-03-11T21:02:17.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-87"}]: dispatch 2024-03-11T21:02:18.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:18 smithi129 ceph-mon[30184]: pgmap v715: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-11T21:02:18.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:18 smithi129 ceph-mon[30184]: osdmap e477: 8 total, 8 up, 8 in 2024-03-11T21:02:18.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:18 smithi129 ceph-mon[42066]: pgmap v715: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-11T21:02:18.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:18 smithi129 ceph-mon[42066]: osdmap e477: 8 total, 8 up, 8 in 2024-03-11T21:02:18.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:18 smithi157 ceph-mon[33359]: pgmap v715: 292 pgs: 292 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-11T21:02:18.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:18 smithi157 ceph-mon[33359]: osdmap e477: 8 total, 8 up, 8 in 2024-03-11T21:02:19.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:19 smithi129 ceph-mon[30184]: osdmap e478: 8 total, 8 up, 8 in 2024-03-11T21:02:19.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:19.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:19.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:19.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:19.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:19 smithi129 ceph-mon[30184]: osdmap e479: 8 total, 8 up, 8 in 2024-03-11T21:02:19.981 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:19.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:19 smithi129 ceph-mon[42066]: osdmap e478: 8 total, 8 up, 8 in 2024-03-11T21:02:19.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:19.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:19.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:19.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:19.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:19 smithi129 ceph-mon[42066]: osdmap e479: 8 total, 8 up, 8 in 2024-03-11T21:02:19.982 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:19.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:19 smithi157 ceph-mon[33359]: osdmap e478: 8 total, 8 up, 8 in 2024-03-11T21:02:19.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:19.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:19.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:19.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:19.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:19 smithi157 ceph-mon[33359]: osdmap e479: 8 total, 8 up, 8 in 2024-03-11T21:02:19.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:20 smithi129 ceph-mon[30184]: pgmap v718: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:02:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:20.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:20.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:02:20.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:20 smithi129 ceph-mon[30184]: osdmap e480: 8 total, 8 up, 8 in 2024-03-11T21:02:20.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:20 smithi129 ceph-mon[42066]: pgmap v718: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:02:20.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:20.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:20.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:02:20.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:20 smithi129 ceph-mon[42066]: osdmap e480: 8 total, 8 up, 8 in 2024-03-11T21:02:20.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:20 smithi157 ceph-mon[33359]: pgmap v718: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:02:20.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:20.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:02:20.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:02:20.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:20 smithi157 ceph-mon[33359]: osdmap e480: 8 total, 8 up, 8 in 2024-03-11T21:02:21.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:21 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:21.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:21 smithi129 ceph-mon[30184]: osdmap e481: 8 total, 8 up, 8 in 2024-03-11T21:02:21.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:21 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:21.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:21 smithi129 ceph-mon[42066]: osdmap e481: 8 total, 8 up, 8 in 2024-03-11T21:02:21.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:21 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:21.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:21 smithi157 ceph-mon[33359]: osdmap e481: 8 total, 8 up, 8 in 2024-03-11T21:02:22.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:22 smithi129 ceph-mon[30184]: pgmap v721: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:22.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:22 smithi129 ceph-mon[30184]: osdmap e482: 8 total, 8 up, 8 in 2024-03-11T21:02:22.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:22.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:22.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-90"}]: dispatch 2024-03-11T21:02:22.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-90"}]: dispatch 2024-03-11T21:02:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:22 smithi129 ceph-mon[42066]: pgmap v721: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:22 smithi129 ceph-mon[42066]: osdmap e482: 8 total, 8 up, 8 in 2024-03-11T21:02:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-90"}]: dispatch 2024-03-11T21:02:22.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-90"}]: dispatch 2024-03-11T21:02:22.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:22 smithi157 ceph-mon[33359]: pgmap v721: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 621 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:22.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:22 smithi157 ceph-mon[33359]: osdmap e482: 8 total, 8 up, 8 in 2024-03-11T21:02:22.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:22.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:22.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-90"}]: dispatch 2024-03-11T21:02:22.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-90"}]: dispatch 2024-03-11T21:02:23.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:02:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:02:24.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:24 smithi129 ceph-mon[30184]: pgmap v724: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-03-11T21:02:24.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:24 smithi129 ceph-mon[30184]: osdmap e483: 8 total, 8 up, 8 in 2024-03-11T21:02:24.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:24 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:24 smithi129 ceph-mon[42066]: pgmap v724: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-03-11T21:02:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:24 smithi129 ceph-mon[42066]: osdmap e483: 8 total, 8 up, 8 in 2024-03-11T21:02:24.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:24 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:24.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:24 smithi157 ceph-mon[33359]: pgmap v724: 292 pgs: 292 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-03-11T21:02:24.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:24 smithi157 ceph-mon[33359]: osdmap e483: 8 total, 8 up, 8 in 2024-03-11T21:02:24.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:24 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:25.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:25 smithi129 ceph-mon[30184]: osdmap e484: 8 total, 8 up, 8 in 2024-03-11T21:02:25.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:25.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:25.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:25 smithi129 ceph-mon[42066]: osdmap e484: 8 total, 8 up, 8 in 2024-03-11T21:02:25.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:25.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:25.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:25 smithi157 ceph-mon[33359]: osdmap e484: 8 total, 8 up, 8 in 2024-03-11T21:02:25.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:25.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:26.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:02:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:02:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:02:26.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:26 smithi129 ceph-mon[30184]: pgmap v727: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:02:26.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:26 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:26.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:26 smithi129 ceph-mon[30184]: osdmap e485: 8 total, 8 up, 8 in 2024-03-11T21:02:26.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:26 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:26.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:26 smithi129 ceph-mon[42066]: pgmap v727: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:02:26.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:26 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:26.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:26 smithi129 ceph-mon[42066]: osdmap e485: 8 total, 8 up, 8 in 2024-03-11T21:02:26.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:26 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:26.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:26 smithi157 ceph-mon[33359]: pgmap v727: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:02:26.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:26 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:26.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:26 smithi157 ceph-mon[33359]: osdmap e485: 8 total, 8 up, 8 in 2024-03-11T21:02:26.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:26 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:27.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:27 smithi129 ceph-mon[30184]: osdmap e486: 8 total, 8 up, 8 in 2024-03-11T21:02:27.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:27 smithi129 ceph-mon[42066]: osdmap e486: 8 total, 8 up, 8 in 2024-03-11T21:02:27.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:27 smithi157 ceph-mon[33359]: osdmap e486: 8 total, 8 up, 8 in 2024-03-11T21:02:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:28 smithi129 ceph-mon[30184]: pgmap v730: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:28 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:28.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:28 smithi129 ceph-mon[30184]: osdmap e487: 8 total, 8 up, 8 in 2024-03-11T21:02:28.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:28.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:28.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-92"}]: dispatch 2024-03-11T21:02:28.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-92"}]: dispatch 2024-03-11T21:02:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:28 smithi129 ceph-mon[42066]: pgmap v730: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:28 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:28 smithi129 ceph-mon[42066]: osdmap e487: 8 total, 8 up, 8 in 2024-03-11T21:02:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:28.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-92"}]: dispatch 2024-03-11T21:02:28.981 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-92"}]: dispatch 2024-03-11T21:02:28.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:28 smithi157 ceph-mon[33359]: pgmap v730: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:28.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:28 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:02:28.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:28 smithi157 ceph-mon[33359]: osdmap e487: 8 total, 8 up, 8 in 2024-03-11T21:02:28.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:28.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:28.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-92"}]: dispatch 2024-03-11T21:02:28.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-92"}]: dispatch 2024-03-11T21:02:29.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:29 smithi129 ceph-mon[30184]: osdmap e488: 8 total, 8 up, 8 in 2024-03-11T21:02:29.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:29 smithi129 ceph-mon[42066]: osdmap e488: 8 total, 8 up, 8 in 2024-03-11T21:02:29.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:29 smithi157 ceph-mon[33359]: osdmap e488: 8 total, 8 up, 8 in 2024-03-11T21:02:30.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:30 smithi129 ceph-mon[30184]: pgmap v733: 260 pgs: 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:02:30.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:30 smithi129 ceph-mon[30184]: osdmap e489: 8 total, 8 up, 8 in 2024-03-11T21:02:30.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:30.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:30 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:30.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:30 smithi129 ceph-mon[42066]: pgmap v733: 260 pgs: 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:02:30.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:30 smithi129 ceph-mon[42066]: osdmap e489: 8 total, 8 up, 8 in 2024-03-11T21:02:30.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:30.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:30 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:30.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:30 smithi157 ceph-mon[33359]: pgmap v733: 260 pgs: 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:02:30.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:30 smithi157 ceph-mon[33359]: osdmap e489: 8 total, 8 up, 8 in 2024-03-11T21:02:30.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:30.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:30 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:31.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:31.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:31 smithi129 ceph-mon[42066]: osdmap e490: 8 total, 8 up, 8 in 2024-03-11T21:02:31.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:31.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:31.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:31 smithi129 ceph-mon[30184]: osdmap e490: 8 total, 8 up, 8 in 2024-03-11T21:02:31.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:31.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:31 smithi157 ceph-mon[33359]: osdmap e490: 8 total, 8 up, 8 in 2024-03-11T21:02:31.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:32.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:32 smithi129 ceph-mon[30184]: pgmap v736: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:32.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:32 smithi129 ceph-mon[30184]: osdmap e491: 8 total, 8 up, 8 in 2024-03-11T21:02:32.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:32 smithi129 ceph-mon[42066]: pgmap v736: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:32.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:32 smithi129 ceph-mon[42066]: osdmap e491: 8 total, 8 up, 8 in 2024-03-11T21:02:32.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:32 smithi157 ceph-mon[33359]: pgmap v736: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:32.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:32 smithi157 ceph-mon[33359]: osdmap e491: 8 total, 8 up, 8 in 2024-03-11T21:02:33.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:33 smithi129 ceph-mon[42066]: osdmap e492: 8 total, 8 up, 8 in 2024-03-11T21:02:33.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:33 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:33.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:33 smithi129 ceph-mon[30184]: osdmap e492: 8 total, 8 up, 8 in 2024-03-11T21:02:33.980 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:33 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:33.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:02:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:02:33.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:33 smithi157 ceph-mon[33359]: osdmap e492: 8 total, 8 up, 8 in 2024-03-11T21:02:33.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:33 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:34.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:34 smithi129 ceph-mon[30184]: pgmap v739: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-11T21:02:34.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:34 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:34.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:34 smithi129 ceph-mon[42066]: pgmap v739: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-11T21:02:34.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:34 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:34.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:34 smithi157 ceph-mon[33359]: pgmap v739: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-11T21:02:34.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:34 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:36.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:02:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:02:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:02:36.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:36 smithi129 ceph-mon[30184]: pgmap v740: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-11T21:02:36.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:36 smithi129 ceph-mon[42066]: pgmap v740: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-11T21:02:36.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:36 smithi157 ceph-mon[33359]: pgmap v740: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-11T21:02:38.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:38 smithi129 ceph-mon[30184]: pgmap v741: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-11T21:02:38.980 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:38 smithi129 ceph-mon[42066]: pgmap v741: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-11T21:02:38.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:38 smithi157 ceph-mon[33359]: pgmap v741: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-11T21:02:40.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:40 smithi129 ceph-mon[30184]: pgmap v742: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-11T21:02:40.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:40 smithi129 ceph-mon[42066]: pgmap v742: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-11T21:02:40.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:40 smithi157 ceph-mon[33359]: pgmap v742: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-11T21:02:42.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:42 smithi129 ceph-mon[30184]: pgmap v743: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 999 B/s wr, 4 op/s 2024-03-11T21:02:42.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:42 smithi129 ceph-mon[42066]: pgmap v743: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 999 B/s wr, 4 op/s 2024-03-11T21:02:42.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:42 smithi157 ceph-mon[33359]: pgmap v743: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 999 B/s wr, 4 op/s 2024-03-11T21:02:44.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:02:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:02:45.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:44 smithi129 ceph-mon[30184]: pgmap v744: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 100 B/s wr, 2 op/s 2024-03-11T21:02:45.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:44 smithi129 ceph-mon[30184]: osdmap e493: 8 total, 8 up, 8 in 2024-03-11T21:02:45.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:44 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:45.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:44 smithi129 ceph-mon[42066]: pgmap v744: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 100 B/s wr, 2 op/s 2024-03-11T21:02:45.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:44 smithi129 ceph-mon[42066]: osdmap e493: 8 total, 8 up, 8 in 2024-03-11T21:02:45.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:44 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:45.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:44 smithi157 ceph-mon[33359]: pgmap v744: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 100 B/s wr, 2 op/s 2024-03-11T21:02:45.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:44 smithi157 ceph-mon[33359]: osdmap e493: 8 total, 8 up, 8 in 2024-03-11T21:02:45.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:44 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:46.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:02:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:02:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:02:47.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:46 smithi129 ceph-mon[30184]: pgmap v746: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-11T21:02:47.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:46 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:02:47.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:46 smithi129 ceph-mon[42066]: pgmap v746: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-11T21:02:47.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:46 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:02:47.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:46 smithi157 ceph-mon[33359]: pgmap v746: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-11T21:02:47.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:46 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:02:48.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:02:48.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:02:48.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:02:48.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:02:48.231 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:02:48.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.231 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:02:48.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:02:48.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:02:48.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:02:48.232 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:47 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:02:48.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:47 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:47 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:47 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:47 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:47 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:47 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:47 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:47 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:02:48.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:47 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:02:48.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:47 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:02:48.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:47 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:02:48.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:47 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:02:48.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:47 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:02:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:48 smithi129 ceph-mon[30184]: pgmap v747: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:02:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:48 smithi129 ceph-mon[30184]: Adjusting osd_memory_target on smithi129 to 3248M 2024-03-11T21:02:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:48 smithi129 ceph-mon[30184]: Adjusting osd_memory_target on smithi157 to 3248M 2024-03-11T21:02:49.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:48 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:48 smithi129 ceph-mon[42066]: pgmap v747: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:02:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:48 smithi129 ceph-mon[42066]: Adjusting osd_memory_target on smithi129 to 3248M 2024-03-11T21:02:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:48 smithi129 ceph-mon[42066]: Adjusting osd_memory_target on smithi157 to 3248M 2024-03-11T21:02:49.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:48 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:49.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:48 smithi157 ceph-mon[33359]: pgmap v747: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:02:49.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:48 smithi157 ceph-mon[33359]: Adjusting osd_memory_target on smithi129 to 3248M 2024-03-11T21:02:49.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:48 smithi157 ceph-mon[33359]: Adjusting osd_memory_target on smithi157 to 3248M 2024-03-11T21:02:49.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:48 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:02:51.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:50 smithi129 ceph-mon[30184]: pgmap v748: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:02:51.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:50 smithi129 ceph-mon[30184]: osdmap e494: 8 total, 8 up, 8 in 2024-03-11T21:02:51.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:50 smithi129 ceph-mon[42066]: pgmap v748: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:02:51.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:50 smithi129 ceph-mon[42066]: osdmap e494: 8 total, 8 up, 8 in 2024-03-11T21:02:51.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:50 smithi157 ceph-mon[33359]: pgmap v748: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:02:51.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:50 smithi157 ceph-mon[33359]: osdmap e494: 8 total, 8 up, 8 in 2024-03-11T21:02:53.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:52 smithi129 ceph-mon[30184]: pgmap v750: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:02:53.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:52 smithi129 ceph-mon[42066]: pgmap v750: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:02:53.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:52 smithi157 ceph-mon[33359]: pgmap v750: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:02:54.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:02:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:02:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:54 smithi129 ceph-mon[30184]: pgmap v751: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:02:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:54 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:55.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-94"}]: dispatch 2024-03-11T21:02:55.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:54 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-94"}]: dispatch 2024-03-11T21:02:55.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:54 smithi129 ceph-mon[42066]: pgmap v751: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:02:55.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:54 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:55.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:55.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:55.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-94"}]: dispatch 2024-03-11T21:02:55.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:54 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-94"}]: dispatch 2024-03-11T21:02:55.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:54 smithi157 ceph-mon[33359]: pgmap v751: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:02:55.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:54 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:02:55.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:55.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:02:55.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-94"}]: dispatch 2024-03-11T21:02:55.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:54 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-94"}]: dispatch 2024-03-11T21:02:56.106 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:55 smithi129 ceph-mon[30184]: osdmap e495: 8 total, 8 up, 8 in 2024-03-11T21:02:56.106 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:55 smithi129 ceph-mon[42066]: osdmap e495: 8 total, 8 up, 8 in 2024-03-11T21:02:56.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:55 smithi157 ceph-mon[33359]: osdmap e495: 8 total, 8 up, 8 in 2024-03-11T21:02:56.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:02:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:02:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:02:57.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:56 smithi129 ceph-mon[30184]: pgmap v753: 260 pgs: 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:57.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:56 smithi129 ceph-mon[30184]: osdmap e496: 8 total, 8 up, 8 in 2024-03-11T21:02:57.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:57.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:56 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:57.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:56 smithi129 ceph-mon[42066]: pgmap v753: 260 pgs: 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:57.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:56 smithi129 ceph-mon[42066]: osdmap e496: 8 total, 8 up, 8 in 2024-03-11T21:02:57.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:57.230 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:56 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:57.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:56 smithi157 ceph-mon[33359]: pgmap v753: 260 pgs: 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:57.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:56 smithi157 ceph-mon[33359]: osdmap e496: 8 total, 8 up, 8 in 2024-03-11T21:02:57.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:57.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:56 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:02:57.237 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 21:02:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=cleanup t=2024-03-11T21:02:56.950552629Z level=info msg="Completed cleanup jobs" duration=3.369719ms 2024-03-11T21:02:57.237 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 21:02:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=sqlstore.transactions t=2024-03-11T21:02:56.95769488Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T21:02:58.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:58.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:57 smithi129 ceph-mon[42066]: osdmap e497: 8 total, 8 up, 8 in 2024-03-11T21:02:58.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:58.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:58.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:57 smithi129 ceph-mon[30184]: osdmap e497: 8 total, 8 up, 8 in 2024-03-11T21:02:58.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:58.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:02:58.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:57 smithi157 ceph-mon[33359]: osdmap e497: 8 total, 8 up, 8 in 2024-03-11T21:02:58.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:02:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:58 smithi129 ceph-mon[30184]: pgmap v756: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:59.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:58 smithi129 ceph-mon[30184]: osdmap e498: 8 total, 8 up, 8 in 2024-03-11T21:02:59.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:58 smithi129 ceph-mon[42066]: pgmap v756: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:59.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:58 smithi129 ceph-mon[42066]: osdmap e498: 8 total, 8 up, 8 in 2024-03-11T21:02:59.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:58 smithi157 ceph-mon[33359]: pgmap v756: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:02:59.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:58 smithi157 ceph-mon[33359]: osdmap e498: 8 total, 8 up, 8 in 2024-03-11T21:03:00.229 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:02:59 smithi129 ceph-mon[30184]: pgmap v758: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:03:00.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:02:59 smithi129 ceph-mon[42066]: pgmap v758: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:03:00.236 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:02:59 smithi157 ceph-mon[33359]: pgmap v758: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:03:02.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:01 smithi129 ceph-mon[30184]: pgmap v759: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 1 op/s 2024-03-11T21:03:02.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:01 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.9", "id": [0, 1]}]: dispatch 2024-03-11T21:03:02.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:01 smithi129 ceph-mon[42066]: pgmap v759: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 1 op/s 2024-03-11T21:03:02.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:01 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.9", "id": [0, 1]}]: dispatch 2024-03-11T21:03:02.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:01 smithi157 ceph-mon[33359]: pgmap v759: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 1 op/s 2024-03-11T21:03:02.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:01 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.9", "id": [0, 1]}]: dispatch 2024-03-11T21:03:03.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:03 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.9", "id": [0, 1]}]': finished 2024-03-11T21:03:03.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:03 smithi129 ceph-mon[30184]: osdmap e499: 8 total, 8 up, 8 in 2024-03-11T21:03:03.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:03 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.9", "id": [0, 1]}]': finished 2024-03-11T21:03:03.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:03 smithi129 ceph-mon[42066]: osdmap e499: 8 total, 8 up, 8 in 2024-03-11T21:03:03.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:03 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "313.9", "id": [0, 1]}]': finished 2024-03-11T21:03:03.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:03 smithi157 ceph-mon[33359]: osdmap e499: 8 total, 8 up, 8 in 2024-03-11T21:03:04.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:03:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:03:04.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:04 smithi157 ceph-mon[33359]: pgmap v761: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 997 B/s wr, 3 op/s 2024-03-11T21:03:04.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:04 smithi157 ceph-mon[33359]: osdmap e500: 8 total, 8 up, 8 in 2024-03-11T21:03:04.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:04.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:04 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:04.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:04 smithi129 ceph-mon[30184]: pgmap v761: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 997 B/s wr, 3 op/s 2024-03-11T21:03:04.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:04 smithi129 ceph-mon[30184]: osdmap e500: 8 total, 8 up, 8 in 2024-03-11T21:03:04.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:04.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:04 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:04.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:04 smithi129 ceph-mon[42066]: pgmap v761: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 997 B/s wr, 3 op/s 2024-03-11T21:03:04.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:04 smithi129 ceph-mon[42066]: osdmap e500: 8 total, 8 up, 8 in 2024-03-11T21:03:04.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:04.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:04 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:06.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:06 smithi129 ceph-mon[30184]: pgmap v763: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 717 B/s rd, 143 B/s wr, 1 op/s 2024-03-11T21:03:06.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:03:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:03:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:03:06.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:06 smithi129 ceph-mon[42066]: pgmap v763: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 717 B/s rd, 143 B/s wr, 1 op/s 2024-03-11T21:03:06.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:06 smithi157 ceph-mon[33359]: pgmap v763: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 717 B/s rd, 143 B/s wr, 1 op/s 2024-03-11T21:03:08.229 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:03:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3[65041]: 2024-03-11T21:03:07.831+0000 7f538e406700 -1 snap_mapper.add_oid found existing snaps mapped on 108:918ee6e9:test-rados-api-smithi129-82818-97::foo:21, removing 2024-03-11T21:03:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:08 smithi129 ceph-mon[42066]: pgmap v764: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s; 1 B/s, 0 objects/s recovering 2024-03-11T21:03:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-96"}]: dispatch 2024-03-11T21:03:08.229 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-96"}]: dispatch 2024-03-11T21:03:08.230 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 21:03:07 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-1[52552]: 2024-03-11T21:03:07.831+0000 7f33ed5e7700 -1 snap_mapper.add_oid found existing snaps mapped on 108:918ee6e9:test-rados-api-smithi129-82818-97::foo:21, removing 2024-03-11T21:03:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:08 smithi129 ceph-mon[30184]: pgmap v764: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s; 1 B/s, 0 objects/s recovering 2024-03-11T21:03:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-96"}]: dispatch 2024-03-11T21:03:08.230 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-96"}]: dispatch 2024-03-11T21:03:08.236 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:03:07 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:03:07.831+0000 7f86d65be700 -1 snap_mapper.add_oid found existing snaps mapped on 108:918ee6e9:test-rados-api-smithi129-82818-97::foo:21, removing 2024-03-11T21:03:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:08 smithi157 ceph-mon[33359]: pgmap v764: 292 pgs: 292 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s; 1 B/s, 0 objects/s recovering 2024-03-11T21:03:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:08.237 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:08.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-96"}]: dispatch 2024-03-11T21:03:08.238 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-96"}]: dispatch 2024-03-11T21:03:09.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:09 smithi129 ceph-mon[30184]: osdmap e501: 8 total, 8 up, 8 in 2024-03-11T21:03:09.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:09 smithi129 ceph-mon[42066]: osdmap e501: 8 total, 8 up, 8 in 2024-03-11T21:03:09.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:09 smithi157 ceph-mon[33359]: osdmap e501: 8 total, 8 up, 8 in 2024-03-11T21:03:10.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:10 smithi129 ceph-mon[30184]: pgmap v766: 260 pgs: 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-03-11T21:03:10.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:10 smithi129 ceph-mon[30184]: osdmap e502: 8 total, 8 up, 8 in 2024-03-11T21:03:10.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:10.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:10.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:10 smithi129 ceph-mon[42066]: pgmap v766: 260 pgs: 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-03-11T21:03:10.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:10 smithi129 ceph-mon[42066]: osdmap e502: 8 total, 8 up, 8 in 2024-03-11T21:03:10.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:10.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:10.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:10 smithi157 ceph-mon[33359]: pgmap v766: 260 pgs: 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-03-11T21:03:10.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:10 smithi157 ceph-mon[33359]: osdmap e502: 8 total, 8 up, 8 in 2024-03-11T21:03:10.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:10.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:11.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:11.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:11 smithi129 ceph-mon[30184]: osdmap e503: 8 total, 8 up, 8 in 2024-03-11T21:03:11.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:03:11.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:03:11.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:11.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:11 smithi129 ceph-mon[42066]: osdmap e503: 8 total, 8 up, 8 in 2024-03-11T21:03:11.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:03:11.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:03:11.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:11.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:11 smithi157 ceph-mon[33359]: osdmap e503: 8 total, 8 up, 8 in 2024-03-11T21:03:11.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:03:11.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:03:12.386 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:12 smithi157 ceph-mon[33359]: pgmap v769: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:03:12.387 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:03:12.387 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:12 smithi157 ceph-mon[33359]: osdmap e504: 8 total, 8 up, 8 in 2024-03-11T21:03:12.387 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-98", "mode": "writeback"}]: dispatch 2024-03-11T21:03:12.387 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-98", "mode": "writeback"}]: dispatch 2024-03-11T21:03:12.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:12 smithi129 ceph-mon[30184]: pgmap v769: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:03:12.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:03:12.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:12 smithi129 ceph-mon[30184]: osdmap e504: 8 total, 8 up, 8 in 2024-03-11T21:03:12.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-98", "mode": "writeback"}]: dispatch 2024-03-11T21:03:12.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-98", "mode": "writeback"}]: dispatch 2024-03-11T21:03:12.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:12 smithi129 ceph-mon[42066]: pgmap v769: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:03:12.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:03:12.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:12 smithi129 ceph-mon[42066]: osdmap e504: 8 total, 8 up, 8 in 2024-03-11T21:03:12.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-98", "mode": "writeback"}]: dispatch 2024-03-11T21:03:12.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-98", "mode": "writeback"}]: dispatch 2024-03-11T21:03:13.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:03:13.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-98", "mode": "writeback"}]': finished 2024-03-11T21:03:13.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[30184]: osdmap e505: 8 total, 8 up, 8 in 2024-03-11T21:03:13.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-98"}]: dispatch 2024-03-11T21:03:13.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-98"}]: dispatch 2024-03-11T21:03:13.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-98"}]': finished 2024-03-11T21:03:13.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[30184]: osdmap e506: 8 total, 8 up, 8 in 2024-03-11T21:03:13.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:03:13.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:03:13.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:03:13.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-98", "mode": "writeback"}]': finished 2024-03-11T21:03:13.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[42066]: osdmap e505: 8 total, 8 up, 8 in 2024-03-11T21:03:13.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-98"}]: dispatch 2024-03-11T21:03:13.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-98"}]: dispatch 2024-03-11T21:03:13.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-98"}]': finished 2024-03-11T21:03:13.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[42066]: osdmap e506: 8 total, 8 up, 8 in 2024-03-11T21:03:13.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:03:13.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:03:13.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:13 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:03:13.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-98", "mode": "writeback"}]': finished 2024-03-11T21:03:13.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:13 smithi157 ceph-mon[33359]: osdmap e505: 8 total, 8 up, 8 in 2024-03-11T21:03:13.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-98"}]: dispatch 2024-03-11T21:03:13.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-98"}]: dispatch 2024-03-11T21:03:13.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-98"}]': finished 2024-03-11T21:03:13.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:13 smithi157 ceph-mon[33359]: osdmap e506: 8 total, 8 up, 8 in 2024-03-11T21:03:13.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:03:13.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:03:14.065 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:03:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:03:14.185 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a533f0 2024-03-11T21:03:14.185 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a527a0 2024-03-11T21:03:14.185 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a528f0 2024-03-11T21:03:14.185 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a52a00 2024-03-11T21:03:14.185 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a53500 2024-03-11T21:03:14.185 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a539d0 2024-03-11T21:03:14.185 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a53b20 2024-03-11T21:03:14.185 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a52f00 2024-03-11T21:03:14.186 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a53030 2024-03-11T21:03:14.186 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a53160 2024-03-11T21:03:14.186 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a53c50 2024-03-11T21:03:14.186 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a54120 2024-03-11T21:03:14.186 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a54250 2024-03-11T21:03:14.186 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a54810 2024-03-11T21:03:14.186 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a53d80 2024-03-11T21:03:14.186 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a53eb0 2024-03-11T21:03:14.186 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a54000 2024-03-11T21:03:14.186 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a54960 2024-03-11T21:03:14.186 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a54360 2024-03-11T21:03:14.186 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a54490 2024-03-11T21:03:14.186 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a545a0 2024-03-11T21:03:14.187 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a546f0 2024-03-11T21:03:14.187 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a54f40 2024-03-11T21:03:14.187 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a54a90 2024-03-11T21:03:14.187 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a54bc0 2024-03-11T21:03:14.187 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a54d10 2024-03-11T21:03:14.187 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a55090 2024-03-11T21:03:14.187 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a55540 2024-03-11T21:03:14.187 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a55650 2024-03-11T21:03:14.187 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a55c50 2024-03-11T21:03:14.187 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a551a0 2024-03-11T21:03:14.187 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a552d0 2024-03-11T21:03:14.187 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a55400 2024-03-11T21:03:14.187 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a55d60 2024-03-11T21:03:14.187 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a557a0 2024-03-11T21:03:14.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a558f0 2024-03-11T21:03:14.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a55a00 2024-03-11T21:03:14.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a55eb0 2024-03-11T21:03:14.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a56360 2024-03-11T21:03:14.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a564b0 2024-03-11T21:03:14.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a56a90 2024-03-11T21:03:14.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a55fc0 2024-03-11T21:03:14.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a560f0 2024-03-11T21:03:14.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a56220 2024-03-11T21:03:14.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a56be0 2024-03-11T21:03:14.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a565e0 2024-03-11T21:03:14.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a56730 2024-03-11T21:03:14.188 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a56840 2024-03-11T21:03:14.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a56950 2024-03-11T21:03:14.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a571c0 2024-03-11T21:03:14.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 0x558254a56d10 2024-03-11T21:03:14.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.189 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.190 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.191 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.191 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.191 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.191 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.191 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.191 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.191 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.191 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.191 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.191 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.191 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.191 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.191 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:03:14.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:14 smithi129 ceph-mon[30184]: pgmap v772: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:03:14.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:14 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:14.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:14 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:03:14.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:03:14.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:14 smithi129 ceph-mon[30184]: osdmap e507: 8 total, 8 up, 8 in 2024-03-11T21:03:14.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-11T21:03:14.480 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:14 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-11T21:03:14.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:14 smithi129 ceph-mon[42066]: pgmap v772: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:03:14.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:14 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:14.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:14 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:03:14.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:03:14.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:14 smithi129 ceph-mon[42066]: osdmap e507: 8 total, 8 up, 8 in 2024-03-11T21:03:14.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-11T21:03:14.481 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:14 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-11T21:03:14.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:14 smithi157 ceph-mon[33359]: pgmap v772: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:03:14.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:14 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:14.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:14 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:03:14.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:03:14.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:14 smithi157 ceph-mon[33359]: osdmap e507: 8 total, 8 up, 8 in 2024-03-11T21:03:14.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-11T21:03:14.489 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:14 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-11T21:03:16.479 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:03:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:03:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:03:16.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:16 smithi129 ceph-mon[30184]: pgmap v775: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:03:16.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:16 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_count","val": "1"}]': finished 2024-03-11T21:03:16.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:16 smithi129 ceph-mon[30184]: osdmap e508: 8 total, 8 up, 8 in 2024-03-11T21:03:16.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:03:16.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:16 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:03:16.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:16 smithi129 ceph-mon[42066]: pgmap v775: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:03:16.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:16 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_count","val": "1"}]': finished 2024-03-11T21:03:16.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:16 smithi129 ceph-mon[42066]: osdmap e508: 8 total, 8 up, 8 in 2024-03-11T21:03:16.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:03:16.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:16 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:03:16.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:16 smithi157 ceph-mon[33359]: pgmap v775: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:03:16.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:16 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_count","val": "1"}]': finished 2024-03-11T21:03:16.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:16 smithi157 ceph-mon[33359]: osdmap e508: 8 total, 8 up, 8 in 2024-03-11T21:03:16.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:03:16.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:16 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:03:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:03:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:17 smithi129 ceph-mon[30184]: osdmap e509: 8 total, 8 up, 8 in 2024-03-11T21:03:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-11T21:03:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-11T21:03:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:17 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "target_max_objects","val": "250"}]': finished 2024-03-11T21:03:17.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:17 smithi129 ceph-mon[30184]: osdmap e510: 8 total, 8 up, 8 in 2024-03-11T21:03:17.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:03:17.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:17 smithi129 ceph-mon[42066]: osdmap e509: 8 total, 8 up, 8 in 2024-03-11T21:03:17.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-11T21:03:17.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-11T21:03:17.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:17 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "target_max_objects","val": "250"}]': finished 2024-03-11T21:03:17.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:17 smithi129 ceph-mon[42066]: osdmap e510: 8 total, 8 up, 8 in 2024-03-11T21:03:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:03:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:17 smithi157 ceph-mon[33359]: osdmap e509: 8 total, 8 up, 8 in 2024-03-11T21:03:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-11T21:03:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-11T21:03:17.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:17 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-98","var": "target_max_objects","val": "250"}]': finished 2024-03-11T21:03:17.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:17 smithi157 ceph-mon[33359]: osdmap e510: 8 total, 8 up, 8 in 2024-03-11T21:03:18.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:18 smithi129 ceph-mon[30184]: pgmap v778: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:18.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:18 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:18.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:18 smithi129 ceph-mon[42066]: pgmap v778: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:18.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:18.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:18 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:18.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:18 smithi157 ceph-mon[33359]: pgmap v778: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:18.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:18.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:18 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:19.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T21:03:19.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:19 smithi129 ceph-mon[30184]: osdmap e511: 8 total, 8 up, 8 in 2024-03-11T21:03:19.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98"}]: dispatch 2024-03-11T21:03:19.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:19 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98"}]: dispatch 2024-03-11T21:03:19.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:19.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T21:03:19.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:19 smithi129 ceph-mon[42066]: osdmap e511: 8 total, 8 up, 8 in 2024-03-11T21:03:19.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98"}]: dispatch 2024-03-11T21:03:19.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:19 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98"}]: dispatch 2024-03-11T21:03:19.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:19.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T21:03:19.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:19 smithi157 ceph-mon[33359]: osdmap e511: 8 total, 8 up, 8 in 2024-03-11T21:03:19.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98"}]: dispatch 2024-03-11T21:03:19.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:19 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98"}]: dispatch 2024-03-11T21:03:19.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:20.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:20 smithi129 ceph-mon[30184]: pgmap v781: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:20.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:20 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98"}]': finished 2024-03-11T21:03:20.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:20 smithi129 ceph-mon[30184]: osdmap e512: 8 total, 8 up, 8 in 2024-03-11T21:03:20.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:20 smithi129 ceph-mon[42066]: pgmap v781: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:20.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:20 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98"}]': finished 2024-03-11T21:03:20.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:20 smithi129 ceph-mon[42066]: osdmap e512: 8 total, 8 up, 8 in 2024-03-11T21:03:20.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:20 smithi157 ceph-mon[33359]: pgmap v781: 292 pgs: 292 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:20.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:20 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-98"}]': finished 2024-03-11T21:03:20.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:20 smithi157 ceph-mon[33359]: osdmap e512: 8 total, 8 up, 8 in 2024-03-11T21:03:21.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:21 smithi129 ceph-mon[30184]: osdmap e513: 8 total, 8 up, 8 in 2024-03-11T21:03:21.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:21 smithi129 ceph-mon[42066]: osdmap e513: 8 total, 8 up, 8 in 2024-03-11T21:03:21.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:21 smithi157 ceph-mon[33359]: osdmap e513: 8 total, 8 up, 8 in 2024-03-11T21:03:22.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:22 smithi129 ceph-mon[30184]: pgmap v784: 260 pgs: 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:03:22.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:22 smithi129 ceph-mon[30184]: osdmap e514: 8 total, 8 up, 8 in 2024-03-11T21:03:22.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:22.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:22 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:22.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:22 smithi129 ceph-mon[42066]: pgmap v784: 260 pgs: 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:03:22.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:22 smithi129 ceph-mon[42066]: osdmap e514: 8 total, 8 up, 8 in 2024-03-11T21:03:22.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:22.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:22 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:22.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:22 smithi157 ceph-mon[33359]: pgmap v784: 260 pgs: 260 active+clean; 8.4 MiB data, 637 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:03:22.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:22 smithi157 ceph-mon[33359]: osdmap e514: 8 total, 8 up, 8 in 2024-03-11T21:03:22.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:22.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:22 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:23.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:23 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:23.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:23 smithi129 ceph-mon[30184]: osdmap e515: 8 total, 8 up, 8 in 2024-03-11T21:03:23.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:03:23.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:23 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:03:23.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:23 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:23.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:23 smithi129 ceph-mon[42066]: osdmap e515: 8 total, 8 up, 8 in 2024-03-11T21:03:23.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:03:23.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:23 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:03:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:23 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:23 smithi157 ceph-mon[33359]: osdmap e515: 8 total, 8 up, 8 in 2024-03-11T21:03:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:03:23.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:23 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:03:24.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:03:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:03:24.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:24 smithi129 ceph-mon[30184]: pgmap v787: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:24.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:03:24.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:24 smithi129 ceph-mon[30184]: osdmap e516: 8 total, 8 up, 8 in 2024-03-11T21:03:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-100"}]: dispatch 2024-03-11T21:03:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:24 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-100"}]: dispatch 2024-03-11T21:03:24.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:24 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:24 smithi129 ceph-mon[42066]: pgmap v787: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:03:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:24 smithi129 ceph-mon[42066]: osdmap e516: 8 total, 8 up, 8 in 2024-03-11T21:03:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-100"}]: dispatch 2024-03-11T21:03:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:24 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-100"}]: dispatch 2024-03-11T21:03:24.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:24 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:24 smithi157 ceph-mon[33359]: pgmap v787: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:03:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:24 smithi157 ceph-mon[33359]: osdmap e516: 8 total, 8 up, 8 in 2024-03-11T21:03:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-100"}]: dispatch 2024-03-11T21:03:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:24 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-100"}]: dispatch 2024-03-11T21:03:24.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:24 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:25.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-100"}]': finished 2024-03-11T21:03:25.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-100", "mode": "writeback"}]: dispatch 2024-03-11T21:03:25.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:25 smithi129 ceph-mon[30184]: osdmap e517: 8 total, 8 up, 8 in 2024-03-11T21:03:25.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-100", "mode": "writeback"}]: dispatch 2024-03-11T21:03:25.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:25 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:03:25.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:25 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-100", "mode": "writeback"}]': finished 2024-03-11T21:03:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-100"}]': finished 2024-03-11T21:03:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-100", "mode": "writeback"}]: dispatch 2024-03-11T21:03:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:25 smithi129 ceph-mon[42066]: osdmap e517: 8 total, 8 up, 8 in 2024-03-11T21:03:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-100", "mode": "writeback"}]: dispatch 2024-03-11T21:03:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:25 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:03:25.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:25 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-100", "mode": "writeback"}]': finished 2024-03-11T21:03:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-6", "overlaypool": "test-rados-api-smithi129-82818-100"}]': finished 2024-03-11T21:03:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-100", "mode": "writeback"}]: dispatch 2024-03-11T21:03:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:25 smithi157 ceph-mon[33359]: osdmap e517: 8 total, 8 up, 8 in 2024-03-11T21:03:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-100", "mode": "writeback"}]: dispatch 2024-03-11T21:03:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:25 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:03:25.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:25 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-100", "mode": "writeback"}]': finished 2024-03-11T21:03:26.379 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:03:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:03:26.380 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-mon[30184]: pgmap v790: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:26.380 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-mon[30184]: osdmap e518: 8 total, 8 up, 8 in 2024-03-11T21:03:26.380 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:03:26.380 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:03:26.380 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:03:26.380 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-mon[30184]: osdmap e519: 8 total, 8 up, 8 in 2024-03-11T21:03:26.380 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:03:26.380 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:03:26.380 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-mon[42066]: pgmap v790: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:26.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-mon[42066]: osdmap e518: 8 total, 8 up, 8 in 2024-03-11T21:03:26.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:03:26.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:03:26.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:03:26.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-mon[42066]: osdmap e519: 8 total, 8 up, 8 in 2024-03-11T21:03:26.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:03:26.381 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:26 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:03:26.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:26 smithi157 ceph-mon[33359]: pgmap v790: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:26.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:26 smithi157 ceph-mon[33359]: osdmap e518: 8 total, 8 up, 8 in 2024-03-11T21:03:26.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:26 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:03:26.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:26 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:03:26.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:26 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:03:26.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:26 smithi157 ceph-mon[33359]: osdmap e519: 8 total, 8 up, 8 in 2024-03-11T21:03:26.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:26 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:03:26.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:26 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:03:28.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:28 smithi129 ceph-mon[42066]: pgmap v793: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:28.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:03:28.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:28 smithi129 ceph-mon[42066]: osdmap e520: 8 total, 8 up, 8 in 2024-03-11T21:03:28.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:03:28.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:03:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:28 smithi129 ceph-mon[30184]: pgmap v793: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:03:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:28 smithi129 ceph-mon[30184]: osdmap e520: 8 total, 8 up, 8 in 2024-03-11T21:03:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:03:28.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:03:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:28 smithi157 ceph-mon[33359]: pgmap v793: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:03:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:28 smithi157 ceph-mon[33359]: osdmap e520: 8 total, 8 up, 8 in 2024-03-11T21:03:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:03:28.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:03:29.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:29 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:03:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:03:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:29 smithi129 ceph-mon[30184]: osdmap e521: 8 total, 8 up, 8 in 2024-03-11T21:03:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-11T21:03:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-11T21:03:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:29 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-11T21:03:29.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:29 smithi129 ceph-mon[30184]: osdmap e522: 8 total, 8 up, 8 in 2024-03-11T21:03:29.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:29 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:03:29.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:03:29.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:29 smithi129 ceph-mon[42066]: osdmap e521: 8 total, 8 up, 8 in 2024-03-11T21:03:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-11T21:03:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-11T21:03:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:29 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-11T21:03:29.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:29 smithi129 ceph-mon[42066]: osdmap e522: 8 total, 8 up, 8 in 2024-03-11T21:03:29.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:29 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:03:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:03:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:29 smithi157 ceph-mon[33359]: osdmap e521: 8 total, 8 up, 8 in 2024-03-11T21:03:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-11T21:03:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-11T21:03:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:29 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-11T21:03:29.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:29 smithi157 ceph-mon[33359]: osdmap e522: 8 total, 8 up, 8 in 2024-03-11T21:03:30.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:30 smithi129 ceph-mon[30184]: pgmap v796: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:30.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:30.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:30 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:30.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:30 smithi129 ceph-mon[42066]: pgmap v796: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:30.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:30.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:30 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:30.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:30 smithi157 ceph-mon[33359]: pgmap v796: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:30.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:30.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:30 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:31.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T21:03:31.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:31 smithi129 ceph-mon[30184]: osdmap e523: 8 total, 8 up, 8 in 2024-03-11T21:03:31.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100"}]: dispatch 2024-03-11T21:03:31.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:31 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100"}]: dispatch 2024-03-11T21:03:31.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T21:03:31.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:31 smithi129 ceph-mon[42066]: osdmap e523: 8 total, 8 up, 8 in 2024-03-11T21:03:31.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100"}]: dispatch 2024-03-11T21:03:31.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:31 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100"}]: dispatch 2024-03-11T21:03:31.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]': finished 2024-03-11T21:03:31.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:31 smithi157 ceph-mon[33359]: osdmap e523: 8 total, 8 up, 8 in 2024-03-11T21:03:31.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100"}]: dispatch 2024-03-11T21:03:31.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:31 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100"}]: dispatch 2024-03-11T21:03:32.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:32 smithi129 ceph-mon[30184]: pgmap v799: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:03:32.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:32 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100"}]': finished 2024-03-11T21:03:32.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:32 smithi129 ceph-mon[30184]: osdmap e524: 8 total, 8 up, 8 in 2024-03-11T21:03:32.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:32 smithi129 ceph-mon[42066]: pgmap v799: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:03:32.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:32 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100"}]': finished 2024-03-11T21:03:32.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:32 smithi129 ceph-mon[42066]: osdmap e524: 8 total, 8 up, 8 in 2024-03-11T21:03:32.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:32 smithi157 ceph-mon[33359]: pgmap v799: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:03:32.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:32 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-100"}]': finished 2024-03-11T21:03:32.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:32 smithi157 ceph-mon[33359]: osdmap e524: 8 total, 8 up, 8 in 2024-03-11T21:03:33.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:33 smithi129 ceph-mon[42066]: osdmap e525: 8 total, 8 up, 8 in 2024-03-11T21:03:33.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:33 smithi129 ceph-mon[30184]: osdmap e525: 8 total, 8 up, 8 in 2024-03-11T21:03:33.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:33 smithi157 ceph-mon[33359]: osdmap e525: 8 total, 8 up, 8 in 2024-03-11T21:03:34.195 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:03:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:03:34.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:34 smithi129 ceph-mon[42066]: pgmap v802: 260 pgs: 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:03:34.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:34 smithi129 ceph-mon[42066]: osdmap e526: 8 total, 8 up, 8 in 2024-03-11T21:03:34.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:34 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:34.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:34 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:34.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:34 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:34.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:34 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:34.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:34 smithi129 ceph-mon[30184]: pgmap v802: 260 pgs: 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:03:34.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:34 smithi129 ceph-mon[30184]: osdmap e526: 8 total, 8 up, 8 in 2024-03-11T21:03:34.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:34 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:34.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:34 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:34.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:34 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:34.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:34 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:34.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:34 smithi157 ceph-mon[33359]: pgmap v802: 260 pgs: 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:03:34.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:34 smithi157 ceph-mon[33359]: osdmap e526: 8 total, 8 up, 8 in 2024-03-11T21:03:34.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:34 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:34.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:34 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:34.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:34 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:34.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:34 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:35.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:35.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[30184]: osdmap e527: 8 total, 8 up, 8 in 2024-03-11T21:03:35.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:03:35.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:03:35.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:03:35.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:03:35.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[30184]: osdmap e528: 8 total, 8 up, 8 in 2024-03-11T21:03:35.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:03:35.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:03:35.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:35.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[42066]: osdmap e527: 8 total, 8 up, 8 in 2024-03-11T21:03:35.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:03:35.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:03:35.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:03:35.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:03:35.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[42066]: osdmap e528: 8 total, 8 up, 8 in 2024-03-11T21:03:35.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:03:35.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:35 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:03:35.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:35 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:35.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:35 smithi157 ceph-mon[33359]: osdmap e527: 8 total, 8 up, 8 in 2024-03-11T21:03:35.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-11T21:03:35.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:03:35.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:35 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-11T21:03:35.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:35 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-11T21:03:35.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:35 smithi157 ceph-mon[33359]: osdmap e528: 8 total, 8 up, 8 in 2024-03-11T21:03:35.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:03:35.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:35 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-11T21:03:36.478 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:03:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:03:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:03:36.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:36 smithi129 ceph-mon[30184]: pgmap v805: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:03:36.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:03:36.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:36 smithi129 ceph-mon[30184]: osdmap e529: 8 total, 8 up, 8 in 2024-03-11T21:03:36.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:03:36.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:03:36.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:36 smithi129 ceph-mon[42066]: pgmap v805: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:03:36.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:03:36.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:36 smithi129 ceph-mon[42066]: osdmap e529: 8 total, 8 up, 8 in 2024-03-11T21:03:36.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:03:36.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:03:36.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:36 smithi157 ceph-mon[33359]: pgmap v805: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:03:36.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-11T21:03:36.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:36 smithi157 ceph-mon[33359]: osdmap e529: 8 total, 8 up, 8 in 2024-03-11T21:03:36.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:03:36.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-11T21:03:38.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:38 smithi129 ceph-mon[30184]: pgmap v808: 292 pgs: 292 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:38.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:03:38.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:38 smithi129 ceph-mon[30184]: osdmap e530: 8 total, 8 up, 8 in 2024-03-11T21:03:38.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:38.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:38.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-102"}]: dispatch 2024-03-11T21:03:38.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-102"}]: dispatch 2024-03-11T21:03:38.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:38 smithi129 ceph-mon[42066]: pgmap v808: 292 pgs: 292 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:38.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:03:38.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:38 smithi129 ceph-mon[42066]: osdmap e530: 8 total, 8 up, 8 in 2024-03-11T21:03:38.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:38.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:38.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-102"}]: dispatch 2024-03-11T21:03:38.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-102"}]: dispatch 2024-03-11T21:03:38.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:38 smithi157 ceph-mon[33359]: pgmap v808: 292 pgs: 292 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:38.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-11T21:03:38.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:38 smithi157 ceph-mon[33359]: osdmap e530: 8 total, 8 up, 8 in 2024-03-11T21:03:38.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:38.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-6"}]: dispatch 2024-03-11T21:03:38.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2576572880' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-102"}]: dispatch 2024-03-11T21:03:38.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-6", "tierpool": "test-rados-api-smithi129-82818-102"}]: dispatch 2024-03-11T21:03:39.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:39 smithi129 ceph-mon[30184]: osdmap e531: 8 total, 8 up, 8 in 2024-03-11T21:03:39.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:39 smithi129 ceph-mon[42066]: osdmap e531: 8 total, 8 up, 8 in 2024-03-11T21:03:39.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:39 smithi157 ceph-mon[33359]: osdmap e531: 8 total, 8 up, 8 in 2024-03-11T21:03:40.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:40 smithi129 ceph-mon[30184]: pgmap v811: 260 pgs: 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:40 smithi129 ceph-mon[30184]: osdmap e532: 8 total, 8 up, 8 in 2024-03-11T21:03:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi129-82818-104"}]: dispatch 2024-03-11T21:03:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi129-82818-104"}]: dispatch 2024-03-11T21:03:40.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi129-82818-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:03:40.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:40 smithi129 ceph-mon[42066]: pgmap v811: 260 pgs: 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:40.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:40 smithi129 ceph-mon[42066]: osdmap e532: 8 total, 8 up, 8 in 2024-03-11T21:03:40.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi129-82818-104"}]: dispatch 2024-03-11T21:03:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi129-82818-104"}]: dispatch 2024-03-11T21:03:40.480 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi129-82818-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:03:40.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:40 smithi157 ceph-mon[33359]: pgmap v811: 260 pgs: 260 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:40.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:40 smithi157 ceph-mon[33359]: osdmap e532: 8 total, 8 up, 8 in 2024-03-11T21:03:40.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi129-82818-104"}]: dispatch 2024-03-11T21:03:40.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi129-82818-104"}]: dispatch 2024-03-11T21:03:40.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi129-82818-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:03:41.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:41 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi129-82818-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:03:41.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:41 smithi129 ceph-mon[30184]: osdmap e533: 8 total, 8 up, 8 in 2024-03-11T21:03:41.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:41 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi129-82818-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi129-82818-104"}]: dispatch 2024-03-11T21:03:41.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:41 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi129-82818-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:03:41.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:41 smithi129 ceph-mon[42066]: osdmap e533: 8 total, 8 up, 8 in 2024-03-11T21:03:41.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:41 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi129-82818-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi129-82818-104"}]: dispatch 2024-03-11T21:03:41.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:41 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi129-82818-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:03:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:41 smithi157 ceph-mon[33359]: osdmap e533: 8 total, 8 up, 8 in 2024-03-11T21:03:41.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:41 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi129-82818-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi129-82818-104"}]: dispatch 2024-03-11T21:03:42.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:42 smithi129 ceph-mon[30184]: pgmap v814: 228 pgs: 228 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:03:42.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:42 smithi129 ceph-mon[30184]: osdmap e534: 8 total, 8 up, 8 in 2024-03-11T21:03:42.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:42 smithi129 ceph-mon[42066]: pgmap v814: 228 pgs: 228 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:03:42.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:42 smithi129 ceph-mon[42066]: osdmap e534: 8 total, 8 up, 8 in 2024-03-11T21:03:42.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:42 smithi157 ceph-mon[33359]: pgmap v814: 228 pgs: 228 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:03:42.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:42 smithi157 ceph-mon[33359]: osdmap e534: 8 total, 8 up, 8 in 2024-03-11T21:03:43.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi129-82818-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi129-82818-104"}]': finished 2024-03-11T21:03:43.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:43 smithi129 ceph-mon[30184]: osdmap e535: 8 total, 8 up, 8 in 2024-03-11T21:03:43.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi129-82818-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi129-82818-104"}]': finished 2024-03-11T21:03:43.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:43 smithi129 ceph-mon[42066]: osdmap e535: 8 total, 8 up, 8 in 2024-03-11T21:03:43.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi129-82818-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi129-82818-104"}]': finished 2024-03-11T21:03:43.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:43 smithi157 ceph-mon[33359]: osdmap e535: 8 total, 8 up, 8 in 2024-03-11T21:03:44.235 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:03:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:03:44.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:44 smithi129 ceph-mon[30184]: pgmap v817: 236 pgs: 8 unknown, 228 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:03:44.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:44 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:03:44.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:44 smithi129 ceph-mon[30184]: osdmap e536: 8 total, 8 up, 8 in 2024-03-11T21:03:44.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:44 smithi129 ceph-mon[42066]: pgmap v817: 236 pgs: 8 unknown, 228 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:03:44.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:44 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:03:44.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:44 smithi129 ceph-mon[42066]: osdmap e536: 8 total, 8 up, 8 in 2024-03-11T21:03:44.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:44 smithi157 ceph-mon[33359]: pgmap v817: 236 pgs: 8 unknown, 228 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:03:44.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:44 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:03:44.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:44 smithi157 ceph-mon[33359]: osdmap e536: 8 total, 8 up, 8 in 2024-03-11T21:03:45.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:45.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:45 smithi129 ceph-mon[30184]: osdmap e537: 8 total, 8 up, 8 in 2024-03-11T21:03:45.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:45.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:45.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:45 smithi129 ceph-mon[30184]: osdmap e538: 8 total, 8 up, 8 in 2024-03-11T21:03:45.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:45.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:45 smithi129 ceph-mon[42066]: osdmap e537: 8 total, 8 up, 8 in 2024-03-11T21:03:45.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:45.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:45.979 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:45 smithi129 ceph-mon[42066]: osdmap e538: 8 total, 8 up, 8 in 2024-03-11T21:03:45.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:45.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:45 smithi157 ceph-mon[33359]: osdmap e537: 8 total, 8 up, 8 in 2024-03-11T21:03:45.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:45.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:45.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:45 smithi157 ceph-mon[33359]: osdmap e538: 8 total, 8 up, 8 in 2024-03-11T21:03:46.478 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:03:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:03:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:03:46.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:46 smithi129 ceph-mon[30184]: pgmap v820: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:46.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:46 smithi129 ceph-mon[30184]: osdmap e539: 8 total, 8 up, 8 in 2024-03-11T21:03:46.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:46.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:46 smithi129 ceph-mon[42066]: pgmap v820: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:46.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:46 smithi129 ceph-mon[42066]: osdmap e539: 8 total, 8 up, 8 in 2024-03-11T21:03:46.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:46 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:46.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:46 smithi157 ceph-mon[33359]: pgmap v820: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:46.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:46 smithi157 ceph-mon[33359]: osdmap e539: 8 total, 8 up, 8 in 2024-03-11T21:03:46.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:46 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:48.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:48 smithi129 ceph-mon[42066]: pgmap v823: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:03:48.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:48.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:48 smithi129 ceph-mon[42066]: osdmap e540: 8 total, 8 up, 8 in 2024-03-11T21:03:48.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-107", "tierpool": "test-rados-api-smithi129-82818-107-cache"}]: dispatch 2024-03-11T21:03:48.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:48 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:03:48.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:48 smithi129 ceph-mon[30184]: pgmap v823: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:03:48.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:48.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:48 smithi129 ceph-mon[30184]: osdmap e540: 8 total, 8 up, 8 in 2024-03-11T21:03:48.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-107", "tierpool": "test-rados-api-smithi129-82818-107-cache"}]: dispatch 2024-03-11T21:03:48.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:48 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:03:48.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:48 smithi157 ceph-mon[33359]: pgmap v823: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:03:48.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:03:48.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:48 smithi157 ceph-mon[33359]: osdmap e540: 8 total, 8 up, 8 in 2024-03-11T21:03:48.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-107", "tierpool": "test-rados-api-smithi129-82818-107-cache"}]: dispatch 2024-03-11T21:03:48.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:48 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:03:49.453 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-107", "tierpool": "test-rados-api-smithi129-82818-107-cache"}]': finished 2024-03-11T21:03:49.453 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:49 smithi157 ceph-mon[33359]: osdmap e541: 8 total, 8 up, 8 in 2024-03-11T21:03:49.454 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-107", "overlaypool": "test-rados-api-smithi129-82818-107-cache"}]: dispatch 2024-03-11T21:03:49.454 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:49 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:49.454 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-107", "overlaypool": "test-rados-api-smithi129-82818-107-cache"}]': finished 2024-03-11T21:03:49.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-107", "tierpool": "test-rados-api-smithi129-82818-107-cache"}]': finished 2024-03-11T21:03:49.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:49 smithi129 ceph-mon[30184]: osdmap e541: 8 total, 8 up, 8 in 2024-03-11T21:03:49.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-107", "overlaypool": "test-rados-api-smithi129-82818-107-cache"}]: dispatch 2024-03-11T21:03:49.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:49 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:49.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-107", "overlaypool": "test-rados-api-smithi129-82818-107-cache"}]': finished 2024-03-11T21:03:49.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-107", "tierpool": "test-rados-api-smithi129-82818-107-cache"}]': finished 2024-03-11T21:03:49.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:49 smithi129 ceph-mon[42066]: osdmap e541: 8 total, 8 up, 8 in 2024-03-11T21:03:49.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-107", "overlaypool": "test-rados-api-smithi129-82818-107-cache"}]: dispatch 2024-03-11T21:03:49.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:49 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:03:49.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-107", "overlaypool": "test-rados-api-smithi129-82818-107-cache"}]': finished 2024-03-11T21:03:50.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:50 smithi129 ceph-mon[30184]: pgmap v826: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:03:50.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:50 smithi129 ceph-mon[30184]: osdmap e542: 8 total, 8 up, 8 in 2024-03-11T21:03:50.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:50 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:03:50.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-107-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:03:50.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:50 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:03:50.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-107-cache", "mode": "writeback"}]': finished 2024-03-11T21:03:50.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:50 smithi129 ceph-mon[30184]: osdmap e543: 8 total, 8 up, 8 in 2024-03-11T21:03:50.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:50 smithi129 ceph-mon[42066]: pgmap v826: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:03:50.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:50 smithi129 ceph-mon[42066]: osdmap e542: 8 total, 8 up, 8 in 2024-03-11T21:03:50.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:50 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:03:50.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-107-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:03:50.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:50 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:03:50.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-107-cache", "mode": "writeback"}]': finished 2024-03-11T21:03:50.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:50 smithi129 ceph-mon[42066]: osdmap e543: 8 total, 8 up, 8 in 2024-03-11T21:03:50.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:50 smithi157 ceph-mon[33359]: pgmap v826: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:03:50.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:50 smithi157 ceph-mon[33359]: osdmap e542: 8 total, 8 up, 8 in 2024-03-11T21:03:50.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:50 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:03:50.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-107-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:03:50.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:50 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:03:50.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-107-cache", "mode": "writeback"}]': finished 2024-03-11T21:03:50.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:50 smithi157 ceph-mon[33359]: osdmap e543: 8 total, 8 up, 8 in 2024-03-11T21:03:51.513 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-107"}]: dispatch 2024-03-11T21:03:51.513 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:51 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:03:51.513 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:51 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:03:51.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-107"}]: dispatch 2024-03-11T21:03:51.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:51 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:03:51.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:51 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:03:51.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-107"}]: dispatch 2024-03-11T21:03:51.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:51 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:03:51.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:51 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:03:52.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:52 smithi129 ceph-mon[30184]: pgmap v829: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:03:52.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-107"}]': finished 2024-03-11T21:03:52.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:52 smithi129 ceph-mon[30184]: osdmap e544: 8 total, 8 up, 8 in 2024-03-11T21:03:52.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:52 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-107", "tierpool": "test-rados-api-smithi129-82818-107-cache"}]: dispatch 2024-03-11T21:03:52.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:52 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:03:52.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:52 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:03:52.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:52 smithi129 ceph-mon[42066]: pgmap v829: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:03:52.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-107"}]': finished 2024-03-11T21:03:52.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:52 smithi129 ceph-mon[42066]: osdmap e544: 8 total, 8 up, 8 in 2024-03-11T21:03:52.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:52 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-107", "tierpool": "test-rados-api-smithi129-82818-107-cache"}]: dispatch 2024-03-11T21:03:52.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:52 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:03:52.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:52 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:03:52.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:52 smithi157 ceph-mon[33359]: pgmap v829: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:03:52.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-107"}]': finished 2024-03-11T21:03:52.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:52 smithi157 ceph-mon[33359]: osdmap e544: 8 total, 8 up, 8 in 2024-03-11T21:03:52.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:52 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-107", "tierpool": "test-rados-api-smithi129-82818-107-cache"}]: dispatch 2024-03-11T21:03:52.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:52 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:03:52.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:52 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:03:53.280 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:53 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:03:53.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:53 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:03:53.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:53 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-107", "tierpool": "test-rados-api-smithi129-82818-107-cache"}]': finished 2024-03-11T21:03:53.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:53 smithi129 ceph-mon[42066]: osdmap e545: 8 total, 8 up, 8 in 2024-03-11T21:03:53.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:53 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:03:53.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:53 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:03:53.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:53 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:03:53.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:53 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-107", "tierpool": "test-rados-api-smithi129-82818-107-cache"}]': finished 2024-03-11T21:03:53.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:53 smithi129 ceph-mon[30184]: osdmap e545: 8 total, 8 up, 8 in 2024-03-11T21:03:53.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:53 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:03:53.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:53 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:03:53.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:53 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:03:53.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:53 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:03:53.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:53 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1572285269' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-107", "tierpool": "test-rados-api-smithi129-82818-107-cache"}]': finished 2024-03-11T21:03:53.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:53 smithi157 ceph-mon[33359]: osdmap e545: 8 total, 8 up, 8 in 2024-03-11T21:03:53.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:53 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:03:53.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:53 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:03:53.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:53 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:03:54.235 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:03:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:03:54.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:54 smithi129 ceph-mon[30184]: pgmap v832: 300 pgs: 300 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:03:54.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:54 smithi129 ceph-mon[30184]: osdmap e546: 8 total, 8 up, 8 in 2024-03-11T21:03:54.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:54 smithi129 ceph-mon[42066]: pgmap v832: 300 pgs: 300 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:03:54.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:54 smithi129 ceph-mon[42066]: osdmap e546: 8 total, 8 up, 8 in 2024-03-11T21:03:54.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:54 smithi157 ceph-mon[33359]: pgmap v832: 300 pgs: 300 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:03:54.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:54 smithi157 ceph-mon[33359]: osdmap e546: 8 total, 8 up, 8 in 2024-03-11T21:03:55.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:55.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:55 smithi129 ceph-mon[30184]: osdmap e547: 8 total, 8 up, 8 in 2024-03-11T21:03:55.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:55.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:55.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:55.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:55.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82818-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:03:55.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:55 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82818-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:03:55.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:55.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:55 smithi129 ceph-mon[42066]: osdmap e547: 8 total, 8 up, 8 in 2024-03-11T21:03:55.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:55.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:55.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:55.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:55.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82818-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:03:55.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:55 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82818-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:03:55.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:03:55.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:55 smithi157 ceph-mon[33359]: osdmap e547: 8 total, 8 up, 8 in 2024-03-11T21:03:55.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:55.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:55.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:55.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:55.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82818-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:03:55.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:55 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82818-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:03:56.477 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:03:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:03:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:03:56.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:56 smithi129 ceph-mon[42066]: pgmap v835: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:56.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:56 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82818-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:03:56.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:56 smithi129 ceph-mon[42066]: osdmap e548: 8 total, 8 up, 8 in 2024-03-11T21:03:56.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82818-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:56.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:56 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82818-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:56.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:56 smithi129 ceph-mon[30184]: pgmap v835: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:56.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:56 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82818-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:03:56.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:56 smithi129 ceph-mon[30184]: osdmap e548: 8 total, 8 up, 8 in 2024-03-11T21:03:56.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82818-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:56.979 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:56 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82818-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:56.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:56 smithi157 ceph-mon[33359]: pgmap v835: 236 pgs: 236 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:56.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:56 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82818-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:03:56.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:56 smithi157 ceph-mon[33359]: osdmap e548: 8 total, 8 up, 8 in 2024-03-11T21:03:56.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82818-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:56.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:56 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82818-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:03:57.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:57 smithi129 ceph-mon[30184]: osdmap e549: 8 total, 8 up, 8 in 2024-03-11T21:03:57.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:57 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82818-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82818-109"}]': finished 2024-03-11T21:03:57.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:57 smithi129 ceph-mon[30184]: osdmap e550: 8 total, 8 up, 8 in 2024-03-11T21:03:57.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:57 smithi129 ceph-mon[42066]: osdmap e549: 8 total, 8 up, 8 in 2024-03-11T21:03:57.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:57 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82818-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82818-109"}]': finished 2024-03-11T21:03:57.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:57 smithi129 ceph-mon[42066]: osdmap e550: 8 total, 8 up, 8 in 2024-03-11T21:03:57.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:57 smithi157 ceph-mon[33359]: osdmap e549: 8 total, 8 up, 8 in 2024-03-11T21:03:57.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:57 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82818-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82818-109"}]': finished 2024-03-11T21:03:57.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:57 smithi157 ceph-mon[33359]: osdmap e550: 8 total, 8 up, 8 in 2024-03-11T21:03:58.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:58 smithi129 ceph-mon[30184]: pgmap v838: 236 pgs: 236 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:58.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:58 smithi129 ceph-mon[30184]: osdmap e551: 8 total, 8 up, 8 in 2024-03-11T21:03:58.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:58.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:03:58 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:58.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:58 smithi129 ceph-mon[42066]: pgmap v838: 236 pgs: 236 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:58.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:58 smithi129 ceph-mon[42066]: osdmap e551: 8 total, 8 up, 8 in 2024-03-11T21:03:58.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:58.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:03:58 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:58.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:58 smithi157 ceph-mon[33359]: pgmap v838: 236 pgs: 236 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:03:58.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:58 smithi157 ceph-mon[33359]: osdmap e551: 8 total, 8 up, 8 in 2024-03-11T21:03:58.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:03:58.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:03:58 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:04:00.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:00 smithi129 ceph-mon[42066]: pgmap v841: 276 pgs: 40 unknown, 236 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:00.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:00 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:04:00.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:00 smithi129 ceph-mon[42066]: osdmap e552: 8 total, 8 up, 8 in 2024-03-11T21:04:00.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:00 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:00.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:00 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:00.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:00 smithi129 ceph-mon[42066]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:00.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:00 smithi129 ceph-mon[30184]: pgmap v841: 276 pgs: 40 unknown, 236 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:00.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:00 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:04:00.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:00 smithi129 ceph-mon[30184]: osdmap e552: 8 total, 8 up, 8 in 2024-03-11T21:04:00.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:00 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:00.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:00 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:00.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:00 smithi129 ceph-mon[30184]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:00.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:00 smithi157 ceph-mon[33359]: pgmap v841: 276 pgs: 40 unknown, 236 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:00.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:00 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:04:00.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:00 smithi157 ceph-mon[33359]: osdmap e552: 8 total, 8 up, 8 in 2024-03-11T21:04:00.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:00 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:00.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:00 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:00.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:00 smithi157 ceph-mon[33359]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:01.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:01 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]': finished 2024-03-11T21:04:01.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:01 smithi129 ceph-mon[30184]: osdmap e553: 8 total, 8 up, 8 in 2024-03-11T21:04:01.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-109", "overlaypool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:01.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:01 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-109", "overlaypool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:01.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:01 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]': finished 2024-03-11T21:04:01.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:01 smithi129 ceph-mon[42066]: osdmap e553: 8 total, 8 up, 8 in 2024-03-11T21:04:01.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-109", "overlaypool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:01.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:01 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-109", "overlaypool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:01.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:01 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]': finished 2024-03-11T21:04:01.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:01 smithi157 ceph-mon[33359]: osdmap e553: 8 total, 8 up, 8 in 2024-03-11T21:04:01.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-109", "overlaypool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:01.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:01 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-109", "overlaypool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:02.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:02 smithi129 ceph-mon[30184]: pgmap v844: 276 pgs: 40 unknown, 236 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:04:02.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:02 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-109", "overlaypool": "test-rados-api-smithi129-82818-109-cache"}]': finished 2024-03-11T21:04:02.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:02 smithi129 ceph-mon[30184]: osdmap e554: 8 total, 8 up, 8 in 2024-03-11T21:04:02.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:02 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-109-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:04:02.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:02 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-109-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:04:02.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:02 smithi129 ceph-mon[42066]: pgmap v844: 276 pgs: 40 unknown, 236 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:04:02.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:02 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-109", "overlaypool": "test-rados-api-smithi129-82818-109-cache"}]': finished 2024-03-11T21:04:02.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:02 smithi129 ceph-mon[42066]: osdmap e554: 8 total, 8 up, 8 in 2024-03-11T21:04:02.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:02 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-109-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:04:02.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:02 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-109-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:04:02.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:02 smithi157 ceph-mon[33359]: pgmap v844: 276 pgs: 40 unknown, 236 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:04:02.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:02 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-109", "overlaypool": "test-rados-api-smithi129-82818-109-cache"}]': finished 2024-03-11T21:04:02.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:02 smithi157 ceph-mon[33359]: osdmap e554: 8 total, 8 up, 8 in 2024-03-11T21:04:02.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:02 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-109-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:04:02.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:02 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-109-cache", "mode": "writeback"}]: dispatch 2024-03-11T21:04:03.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:03 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:04:03.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-109-cache", "mode": "writeback"}]': finished 2024-03-11T21:04:03.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:03 smithi129 ceph-mon[30184]: osdmap e555: 8 total, 8 up, 8 in 2024-03-11T21:04:03.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:04:03.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:04:03.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:03 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:04:03.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:03 smithi129 ceph-mon[30184]: osdmap e556: 8 total, 8 up, 8 in 2024-03-11T21:04:03.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:04:03.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:03 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:04:03.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-109-cache", "mode": "writeback"}]': finished 2024-03-11T21:04:03.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:03 smithi129 ceph-mon[42066]: osdmap e555: 8 total, 8 up, 8 in 2024-03-11T21:04:03.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:04:03.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:04:03.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:03 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:04:03.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:03 smithi129 ceph-mon[42066]: osdmap e556: 8 total, 8 up, 8 in 2024-03-11T21:04:03.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:04:03.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:03 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:04:03.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-109-cache", "mode": "writeback"}]': finished 2024-03-11T21:04:03.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:03 smithi157 ceph-mon[33359]: osdmap e555: 8 total, 8 up, 8 in 2024-03-11T21:04:03.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:04:03.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:04:03.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:03 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:04:03.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:03 smithi157 ceph-mon[33359]: osdmap e556: 8 total, 8 up, 8 in 2024-03-11T21:04:03.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:04:04.235 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:04:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:04:04.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:04 smithi129 ceph-mon[30184]: pgmap v847: 276 pgs: 276 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:04.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:04:04.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:04:04.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:04.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:04:04.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:04 smithi129 ceph-mon[30184]: osdmap e557: 8 total, 8 up, 8 in 2024-03-11T21:04:04.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:04:04.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:04 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:04:04.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:04 smithi129 ceph-mon[42066]: pgmap v847: 276 pgs: 276 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:04.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:04:04.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:04:04.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:04.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:04:04.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:04 smithi129 ceph-mon[42066]: osdmap e557: 8 total, 8 up, 8 in 2024-03-11T21:04:04.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:04:04.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:04 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:04:04.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:04 smithi157 ceph-mon[33359]: pgmap v847: 276 pgs: 276 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:04.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:04:04.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:04:04.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:04.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:04:04.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:04 smithi157 ceph-mon[33359]: osdmap e557: 8 total, 8 up, 8 in 2024-03-11T21:04:04.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:04:04.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:04 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:04:05.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:05.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:05 smithi129 ceph-mon[30184]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:05.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:05 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:04:05.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:05 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:04:05.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:05 smithi129 ceph-mon[30184]: osdmap e558: 8 total, 8 up, 8 in 2024-03-11T21:04:05.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-11T21:04:05.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:05 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-11T21:04:05.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:05.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:05 smithi129 ceph-mon[42066]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:05.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:05 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:04:05.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:05 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:04:05.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:05 smithi129 ceph-mon[42066]: osdmap e558: 8 total, 8 up, 8 in 2024-03-11T21:04:05.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-11T21:04:05.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:05 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-11T21:04:05.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:05.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:05 smithi157 ceph-mon[33359]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:05.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:05 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:04:05.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:05 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:04:05.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:05 smithi157 ceph-mon[33359]: osdmap e558: 8 total, 8 up, 8 in 2024-03-11T21:04:05.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-11T21:04:05.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:05 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-11T21:04:06.477 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:04:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:04:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:04:06.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:06 smithi129 ceph-mon[30184]: pgmap v850: 276 pgs: 276 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:06.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:06 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-11T21:04:06.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:06 smithi129 ceph-mon[30184]: osdmap e559: 8 total, 8 up, 8 in 2024-03-11T21:04:06.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:06 smithi129 ceph-mon[42066]: pgmap v850: 276 pgs: 276 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:06.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:06 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-11T21:04:06.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:06 smithi129 ceph-mon[42066]: osdmap e559: 8 total, 8 up, 8 in 2024-03-11T21:04:06.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:06 smithi157 ceph-mon[33359]: pgmap v850: 276 pgs: 276 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:06.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:06 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-11T21:04:06.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:06 smithi157 ceph-mon[33359]: osdmap e559: 8 total, 8 up, 8 in 2024-03-11T21:04:07.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:07.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:07 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:07.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:07.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:07 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:07.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:07.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:07 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:08.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:08 smithi129 ceph-mon[30184]: pgmap v853: 276 pgs: 276 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:04:08.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-109"}]': finished 2024-03-11T21:04:08.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:08 smithi129 ceph-mon[30184]: osdmap e560: 8 total, 8 up, 8 in 2024-03-11T21:04:08.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:08.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:08 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:08.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:08 smithi129 ceph-mon[42066]: pgmap v853: 276 pgs: 276 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:04:08.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-109"}]': finished 2024-03-11T21:04:08.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:08 smithi129 ceph-mon[42066]: osdmap e560: 8 total, 8 up, 8 in 2024-03-11T21:04:08.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:08.729 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:08 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:08.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:08 smithi157 ceph-mon[33359]: pgmap v853: 276 pgs: 276 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:04:08.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-109"}]': finished 2024-03-11T21:04:08.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:08 smithi157 ceph-mon[33359]: osdmap e560: 8 total, 8 up, 8 in 2024-03-11T21:04:08.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:08.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:08 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]: dispatch 2024-03-11T21:04:09.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:09 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]': finished 2024-03-11T21:04:09.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:09 smithi129 ceph-mon[30184]: osdmap e561: 8 total, 8 up, 8 in 2024-03-11T21:04:09.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:09 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]': finished 2024-03-11T21:04:09.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:09 smithi129 ceph-mon[42066]: osdmap e561: 8 total, 8 up, 8 in 2024-03-11T21:04:09.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:09 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-109", "tierpool": "test-rados-api-smithi129-82818-109-cache"}]': finished 2024-03-11T21:04:09.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:09 smithi157 ceph-mon[33359]: osdmap e561: 8 total, 8 up, 8 in 2024-03-11T21:04:10.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:10 smithi129 ceph-mon[30184]: pgmap v856: 276 pgs: 276 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:04:10.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:10 smithi129 ceph-mon[30184]: osdmap e562: 8 total, 8 up, 8 in 2024-03-11T21:04:10.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:10 smithi129 ceph-mon[30184]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:10.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:10 smithi129 ceph-mon[42066]: pgmap v856: 276 pgs: 276 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:04:10.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:10 smithi129 ceph-mon[42066]: osdmap e562: 8 total, 8 up, 8 in 2024-03-11T21:04:10.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:10 smithi129 ceph-mon[42066]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:10.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:10 smithi157 ceph-mon[33359]: pgmap v856: 276 pgs: 276 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:04:10.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:10 smithi157 ceph-mon[33359]: osdmap e562: 8 total, 8 up, 8 in 2024-03-11T21:04:10.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:10 smithi157 ceph-mon[33359]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:11.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:11 smithi129 ceph-mon[30184]: osdmap e563: 8 total, 8 up, 8 in 2024-03-11T21:04:11.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:11.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:11 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:11.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:11 smithi129 ceph-mon[42066]: osdmap e563: 8 total, 8 up, 8 in 2024-03-11T21:04:11.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:11.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:11 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:11.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:11 smithi157 ceph-mon[33359]: osdmap e563: 8 total, 8 up, 8 in 2024-03-11T21:04:11.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:11.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:11 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:12.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:12 smithi129 ceph-mon[30184]: pgmap v859: 236 pgs: 236 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:04:12.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-109"}]': finished 2024-03-11T21:04:12.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:12 smithi129 ceph-mon[30184]: osdmap e564: 8 total, 8 up, 8 in 2024-03-11T21:04:12.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:12.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:12.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:12 smithi129 ceph-mon[42066]: pgmap v859: 236 pgs: 236 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:04:12.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-109"}]': finished 2024-03-11T21:04:12.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:12 smithi129 ceph-mon[42066]: osdmap e564: 8 total, 8 up, 8 in 2024-03-11T21:04:12.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:12.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:12.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:12 smithi157 ceph-mon[33359]: pgmap v859: 236 pgs: 236 active+clean; 582 KiB data, 606 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:04:12.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-109"}]': finished 2024-03-11T21:04:12.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:12 smithi157 ceph-mon[33359]: osdmap e564: 8 total, 8 up, 8 in 2024-03-11T21:04:12.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1381193527' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:12.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-109"}]: dispatch 2024-03-11T21:04:13.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-109"}]': finished 2024-03-11T21:04:13.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:13 smithi129 ceph-mon[30184]: osdmap e565: 8 total, 8 up, 8 in 2024-03-11T21:04:13.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-109"}]': finished 2024-03-11T21:04:13.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:13 smithi129 ceph-mon[42066]: osdmap e565: 8 total, 8 up, 8 in 2024-03-11T21:04:13.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-109"}]': finished 2024-03-11T21:04:13.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:13 smithi157 ceph-mon[33359]: osdmap e565: 8 total, 8 up, 8 in 2024-03-11T21:04:14.235 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:04:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:04:14.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:14 smithi129 ceph-mon[30184]: pgmap v862: 236 pgs: 236 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:04:14.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:14 smithi129 ceph-mon[30184]: osdmap e566: 8 total, 8 up, 8 in 2024-03-11T21:04:14.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi129-82818-104"}]: dispatch 2024-03-11T21:04:14.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:14 smithi129 ceph-mon[42066]: pgmap v862: 236 pgs: 236 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:04:14.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:14 smithi129 ceph-mon[42066]: osdmap e566: 8 total, 8 up, 8 in 2024-03-11T21:04:14.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi129-82818-104"}]: dispatch 2024-03-11T21:04:14.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:14 smithi157 ceph-mon[33359]: pgmap v862: 236 pgs: 236 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:04:14.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:14 smithi157 ceph-mon[33359]: osdmap e566: 8 total, 8 up, 8 in 2024-03-11T21:04:14.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi129-82818-104"}]: dispatch 2024-03-11T21:04:15.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:15.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi129-82818-104"}]': finished 2024-03-11T21:04:15.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:15 smithi129 ceph-mon[30184]: osdmap e567: 8 total, 8 up, 8 in 2024-03-11T21:04:15.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi129-82818-104"}]: dispatch 2024-03-11T21:04:15.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:15 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:15.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi129-82818-104"}]': finished 2024-03-11T21:04:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi129-82818-104"}]': finished 2024-03-11T21:04:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:15 smithi129 ceph-mon[42066]: osdmap e567: 8 total, 8 up, 8 in 2024-03-11T21:04:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi129-82818-104"}]: dispatch 2024-03-11T21:04:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:15 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:15.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi129-82818-104"}]': finished 2024-03-11T21:04:15.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:15.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi129-82818-104"}]': finished 2024-03-11T21:04:15.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:15 smithi157 ceph-mon[33359]: osdmap e567: 8 total, 8 up, 8 in 2024-03-11T21:04:15.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi129-82818-104"}]: dispatch 2024-03-11T21:04:15.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:15 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:15.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2114629630' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi129-82818-104"}]': finished 2024-03-11T21:04:16.436 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:04:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:04:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:04:16.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:16 smithi129 ceph-mon[30184]: pgmap v865: 228 pgs: 228 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:04:16.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:16 smithi129 ceph-mon[30184]: osdmap e568: 8 total, 8 up, 8 in 2024-03-11T21:04:16.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:16.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:16.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82818-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:04:16.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:16 smithi129 ceph-mon[42066]: pgmap v865: 228 pgs: 228 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:04:16.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:16 smithi129 ceph-mon[42066]: osdmap e568: 8 total, 8 up, 8 in 2024-03-11T21:04:16.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:16.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:16.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82818-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:04:16.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:16 smithi157 ceph-mon[33359]: pgmap v865: 228 pgs: 228 active+clean; 582 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:04:16.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:16 smithi157 ceph-mon[33359]: osdmap e568: 8 total, 8 up, 8 in 2024-03-11T21:04:16.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:16.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:16.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82818-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-11T21:04:17.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82818-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:04:17.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:17 smithi129 ceph-mon[30184]: osdmap e569: 8 total, 8 up, 8 in 2024-03-11T21:04:17.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82818-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:17.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82818-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:04:17.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:17 smithi129 ceph-mon[42066]: osdmap e569: 8 total, 8 up, 8 in 2024-03-11T21:04:17.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82818-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:17.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi129-82818-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-11T21:04:17.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:17 smithi157 ceph-mon[33359]: osdmap e569: 8 total, 8 up, 8 in 2024-03-11T21:04:17.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82818-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:18.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:18 smithi129 ceph-mon[30184]: pgmap v868: 228 pgs: 228 active+clean; 582 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:04:18.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:18 smithi129 ceph-mon[30184]: osdmap e570: 8 total, 8 up, 8 in 2024-03-11T21:04:18.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:18 smithi129 ceph-mon[42066]: pgmap v868: 228 pgs: 228 active+clean; 582 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:04:18.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:18 smithi129 ceph-mon[42066]: osdmap e570: 8 total, 8 up, 8 in 2024-03-11T21:04:18.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:18 smithi157 ceph-mon[33359]: pgmap v868: 228 pgs: 228 active+clean; 582 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-11T21:04:18.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:18 smithi157 ceph-mon[33359]: osdmap e570: 8 total, 8 up, 8 in 2024-03-11T21:04:19.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82818-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:04:19.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:19 smithi129 ceph-mon[30184]: osdmap e571: 8 total, 8 up, 8 in 2024-03-11T21:04:19.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:04:19.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:19.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82818-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:04:19.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:19 smithi129 ceph-mon[42066]: osdmap e571: 8 total, 8 up, 8 in 2024-03-11T21:04:19.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:04:19.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:19.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi129-82818-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:04:19.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:19 smithi157 ceph-mon[33359]: osdmap e571: 8 total, 8 up, 8 in 2024-03-11T21:04:19.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:04:19.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:20.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:20 smithi129 ceph-mon[42066]: pgmap v871: 236 pgs: 8 unknown, 228 active+clean; 582 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:20.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:20 smithi129 ceph-mon[42066]: osdmap e572: 8 total, 8 up, 8 in 2024-03-11T21:04:20.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:04:20.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:04:20.728 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:20 smithi129 ceph-mon[42066]: osdmap e573: 8 total, 8 up, 8 in 2024-03-11T21:04:20.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:20 smithi129 ceph-mon[30184]: pgmap v871: 236 pgs: 8 unknown, 228 active+clean; 582 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:20.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:20 smithi129 ceph-mon[30184]: osdmap e572: 8 total, 8 up, 8 in 2024-03-11T21:04:20.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:04:20.728 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:04:20.729 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:20 smithi129 ceph-mon[30184]: osdmap e573: 8 total, 8 up, 8 in 2024-03-11T21:04:20.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:20 smithi157 ceph-mon[33359]: pgmap v871: 236 pgs: 8 unknown, 228 active+clean; 582 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:20.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:20 smithi157 ceph-mon[33359]: osdmap e572: 8 total, 8 up, 8 in 2024-03-11T21:04:20.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:04:20.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:04:20.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:20 smithi157 ceph-mon[33359]: osdmap e573: 8 total, 8 up, 8 in 2024-03-11T21:04:21.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:04:21.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:04:21.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:04:22.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:22 smithi129 ceph-mon[30184]: pgmap v874: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:04:22.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:04:22.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:22 smithi129 ceph-mon[30184]: osdmap e574: 8 total, 8 up, 8 in 2024-03-11T21:04:22.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-112"}]: dispatch 2024-03-11T21:04:22.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:22 smithi129 ceph-mon[42066]: pgmap v874: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:04:22.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:04:22.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:22 smithi129 ceph-mon[42066]: osdmap e574: 8 total, 8 up, 8 in 2024-03-11T21:04:22.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-112"}]: dispatch 2024-03-11T21:04:22.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:22 smithi157 ceph-mon[33359]: pgmap v874: 268 pgs: 40 unknown, 228 active+clean; 582 KiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:04:22.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:04:22.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:22 smithi157 ceph-mon[33359]: osdmap e574: 8 total, 8 up, 8 in 2024-03-11T21:04:22.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-112"}]: dispatch 2024-03-11T21:04:23.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-112"}]': finished 2024-03-11T21:04:23.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:23 smithi157 ceph-mon[33359]: osdmap e575: 8 total, 8 up, 8 in 2024-03-11T21:04:23.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:23.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-112"}]': finished 2024-03-11T21:04:23.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:23 smithi129 ceph-mon[30184]: osdmap e575: 8 total, 8 up, 8 in 2024-03-11T21:04:23.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:23.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-112"}]': finished 2024-03-11T21:04:23.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:23 smithi129 ceph-mon[42066]: osdmap e575: 8 total, 8 up, 8 in 2024-03-11T21:04:23.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:24.235 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:04:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:04:24.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:24 smithi129 ceph-mon[30184]: pgmap v877: 268 pgs: 268 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:04:24.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:04:24.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:24 smithi129 ceph-mon[30184]: osdmap e576: 8 total, 8 up, 8 in 2024-03-11T21:04:24.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:24 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-112"}]: dispatch 2024-03-11T21:04:24.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:24 smithi129 ceph-mon[42066]: pgmap v877: 268 pgs: 268 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:04:24.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:04:24.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:24 smithi129 ceph-mon[42066]: osdmap e576: 8 total, 8 up, 8 in 2024-03-11T21:04:24.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:24 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-112"}]: dispatch 2024-03-11T21:04:24.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:24 smithi157 ceph-mon[33359]: pgmap v877: 268 pgs: 268 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:04:24.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:04:24.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:24 smithi157 ceph-mon[33359]: osdmap e576: 8 total, 8 up, 8 in 2024-03-11T21:04:24.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:24 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-112"}]: dispatch 2024-03-11T21:04:25.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:25.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-112"}]': finished 2024-03-11T21:04:25.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:25 smithi129 ceph-mon[30184]: osdmap e577: 8 total, 8 up, 8 in 2024-03-11T21:04:25.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:25.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-112"}]': finished 2024-03-11T21:04:25.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:25 smithi129 ceph-mon[42066]: osdmap e577: 8 total, 8 up, 8 in 2024-03-11T21:04:25.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:25.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-112"}]': finished 2024-03-11T21:04:25.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:25 smithi157 ceph-mon[33359]: osdmap e577: 8 total, 8 up, 8 in 2024-03-11T21:04:26.477 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:04:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:04:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:04:26.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:26 smithi129 ceph-mon[30184]: pgmap v880: 268 pgs: 268 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:04:26.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:26 smithi129 ceph-mon[30184]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:26.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:26 smithi129 ceph-mon[30184]: osdmap e578: 8 total, 8 up, 8 in 2024-03-11T21:04:26.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:26 smithi129 ceph-mon[42066]: pgmap v880: 268 pgs: 268 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:04:26.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:26 smithi129 ceph-mon[42066]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:26.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:26 smithi129 ceph-mon[42066]: osdmap e578: 8 total, 8 up, 8 in 2024-03-11T21:04:26.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:26 smithi157 ceph-mon[33359]: pgmap v880: 268 pgs: 268 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:04:26.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:26 smithi157 ceph-mon[33359]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:26.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:26 smithi157 ceph-mon[33359]: osdmap e578: 8 total, 8 up, 8 in 2024-03-11T21:04:27.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:27 smithi129 ceph-mon[30184]: osdmap e579: 8 total, 8 up, 8 in 2024-03-11T21:04:27.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:04:27.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:04:27.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:27 smithi129 ceph-mon[30184]: osdmap e580: 8 total, 8 up, 8 in 2024-03-11T21:04:27.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:27 smithi129 ceph-mon[42066]: osdmap e579: 8 total, 8 up, 8 in 2024-03-11T21:04:27.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:04:27.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:04:27.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:27 smithi129 ceph-mon[42066]: osdmap e580: 8 total, 8 up, 8 in 2024-03-11T21:04:27.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:27 smithi157 ceph-mon[33359]: osdmap e579: 8 total, 8 up, 8 in 2024-03-11T21:04:27.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:04:27.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:04:27.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:27 smithi157 ceph-mon[33359]: osdmap e580: 8 total, 8 up, 8 in 2024-03-11T21:04:28.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:28 smithi129 ceph-mon[30184]: pgmap v883: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:04:28.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:04:28.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:04:28.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:28 smithi129 ceph-mon[30184]: osdmap e581: 8 total, 8 up, 8 in 2024-03-11T21:04:28.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-114"}]: dispatch 2024-03-11T21:04:28.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:28 smithi129 ceph-mon[42066]: pgmap v883: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:04:28.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:04:28.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:04:28.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:28 smithi129 ceph-mon[42066]: osdmap e581: 8 total, 8 up, 8 in 2024-03-11T21:04:28.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-114"}]: dispatch 2024-03-11T21:04:28.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:28 smithi157 ceph-mon[33359]: pgmap v883: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:04:28.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:04:28.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:04:28.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:28 smithi157 ceph-mon[33359]: osdmap e581: 8 total, 8 up, 8 in 2024-03-11T21:04:28.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-114"}]: dispatch 2024-03-11T21:04:30.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:30 smithi129 ceph-mon[30184]: pgmap v886: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:04:30.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-114"}]': finished 2024-03-11T21:04:30.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:30 smithi129 ceph-mon[30184]: osdmap e582: 8 total, 8 up, 8 in 2024-03-11T21:04:30.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-114", "mode": "writeback"}]: dispatch 2024-03-11T21:04:30.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:30 smithi129 ceph-mon[42066]: pgmap v886: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:04:30.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-114"}]': finished 2024-03-11T21:04:30.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:30 smithi129 ceph-mon[42066]: osdmap e582: 8 total, 8 up, 8 in 2024-03-11T21:04:30.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-114", "mode": "writeback"}]: dispatch 2024-03-11T21:04:30.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:30 smithi157 ceph-mon[33359]: pgmap v886: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:04:30.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-114"}]': finished 2024-03-11T21:04:30.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:30 smithi157 ceph-mon[33359]: osdmap e582: 8 total, 8 up, 8 in 2024-03-11T21:04:30.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-114", "mode": "writeback"}]: dispatch 2024-03-11T21:04:31.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:31 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:04:31.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-114", "mode": "writeback"}]': finished 2024-03-11T21:04:31.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:31 smithi129 ceph-mon[30184]: osdmap e583: 8 total, 8 up, 8 in 2024-03-11T21:04:31.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:31.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:31 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:04:31.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-114", "mode": "writeback"}]': finished 2024-03-11T21:04:31.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:31 smithi129 ceph-mon[42066]: osdmap e583: 8 total, 8 up, 8 in 2024-03-11T21:04:31.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:31.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:31 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:04:31.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-114", "mode": "writeback"}]': finished 2024-03-11T21:04:31.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:31 smithi157 ceph-mon[33359]: osdmap e583: 8 total, 8 up, 8 in 2024-03-11T21:04:31.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:32.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:32 smithi129 ceph-mon[30184]: pgmap v889: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:04:32.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:32 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:04:32.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:32 smithi129 ceph-mon[30184]: osdmap e584: 8 total, 8 up, 8 in 2024-03-11T21:04:32.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:32 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-114"}]: dispatch 2024-03-11T21:04:32.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:32 smithi129 ceph-mon[42066]: pgmap v889: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:04:32.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:32 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:04:32.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:32 smithi129 ceph-mon[42066]: osdmap e584: 8 total, 8 up, 8 in 2024-03-11T21:04:32.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:32 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-114"}]: dispatch 2024-03-11T21:04:32.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:32 smithi157 ceph-mon[33359]: pgmap v889: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:04:32.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:32 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:04:32.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:32 smithi157 ceph-mon[33359]: osdmap e584: 8 total, 8 up, 8 in 2024-03-11T21:04:32.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:32 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-114"}]: dispatch 2024-03-11T21:04:33.832 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:33 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:04:33.833 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-114"}]': finished 2024-03-11T21:04:33.833 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:33 smithi157 ceph-mon[33359]: osdmap e585: 8 total, 8 up, 8 in 2024-03-11T21:04:33.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:33 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:04:33.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-114"}]': finished 2024-03-11T21:04:33.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:33 smithi129 ceph-mon[30184]: osdmap e585: 8 total, 8 up, 8 in 2024-03-11T21:04:33.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:33 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:04:33.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-114"}]': finished 2024-03-11T21:04:33.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:33 smithi129 ceph-mon[42066]: osdmap e585: 8 total, 8 up, 8 in 2024-03-11T21:04:34.234 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:04:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:04:34.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:34 smithi129 ceph-mon[30184]: pgmap v892: 268 pgs: 268 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:04:34.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:34 smithi129 ceph-mon[30184]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:34.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:34 smithi129 ceph-mon[30184]: osdmap e586: 8 total, 8 up, 8 in 2024-03-11T21:04:34.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:34 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:34.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:34 smithi129 ceph-mon[42066]: pgmap v892: 268 pgs: 268 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:04:34.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:34 smithi129 ceph-mon[42066]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:34.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:34 smithi129 ceph-mon[42066]: osdmap e586: 8 total, 8 up, 8 in 2024-03-11T21:04:34.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:34 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:34.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:34 smithi157 ceph-mon[33359]: pgmap v892: 268 pgs: 268 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:04:34.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:34 smithi157 ceph-mon[33359]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:34.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:34 smithi157 ceph-mon[33359]: osdmap e586: 8 total, 8 up, 8 in 2024-03-11T21:04:34.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:34 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:35.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:35.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:35 smithi129 ceph-mon[30184]: osdmap e587: 8 total, 8 up, 8 in 2024-03-11T21:04:35.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:04:35.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:35.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:35 smithi129 ceph-mon[42066]: osdmap e587: 8 total, 8 up, 8 in 2024-03-11T21:04:35.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:04:35.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:35.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:35 smithi157 ceph-mon[33359]: osdmap e587: 8 total, 8 up, 8 in 2024-03-11T21:04:35.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:04:36.477 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:04:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:04:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:04:36.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:36 smithi129 ceph-mon[30184]: pgmap v895: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:04:36.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:04:36.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:36 smithi129 ceph-mon[30184]: osdmap e588: 8 total, 8 up, 8 in 2024-03-11T21:04:36.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:36 smithi129 ceph-mon[42066]: pgmap v895: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:04:36.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:04:36.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:36 smithi129 ceph-mon[42066]: osdmap e588: 8 total, 8 up, 8 in 2024-03-11T21:04:36.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:36 smithi157 ceph-mon[33359]: pgmap v895: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:04:36.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:04:36.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:36 smithi157 ceph-mon[33359]: osdmap e588: 8 total, 8 up, 8 in 2024-03-11T21:04:37.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:37 smithi129 ceph-mon[30184]: osdmap e589: 8 total, 8 up, 8 in 2024-03-11T21:04:37.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:04:37.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:37 smithi129 ceph-mon[42066]: osdmap e589: 8 total, 8 up, 8 in 2024-03-11T21:04:37.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:04:37.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:37 smithi157 ceph-mon[33359]: osdmap e589: 8 total, 8 up, 8 in 2024-03-11T21:04:37.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:04:38.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:38 smithi129 ceph-mon[30184]: pgmap v898: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-11T21:04:38.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:04:38.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:38 smithi129 ceph-mon[30184]: osdmap e590: 8 total, 8 up, 8 in 2024-03-11T21:04:38.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-116"}]: dispatch 2024-03-11T21:04:38.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:38 smithi129 ceph-mon[42066]: pgmap v898: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-11T21:04:38.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:04:38.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:38 smithi129 ceph-mon[42066]: osdmap e590: 8 total, 8 up, 8 in 2024-03-11T21:04:38.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-116"}]: dispatch 2024-03-11T21:04:38.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:38 smithi157 ceph-mon[33359]: pgmap v898: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-11T21:04:38.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:04:38.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:38 smithi157 ceph-mon[33359]: osdmap e590: 8 total, 8 up, 8 in 2024-03-11T21:04:38.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-116"}]: dispatch 2024-03-11T21:04:39.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-116"}]': finished 2024-03-11T21:04:39.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:39 smithi129 ceph-mon[30184]: osdmap e591: 8 total, 8 up, 8 in 2024-03-11T21:04:39.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-116", "mode": "writeback"}]: dispatch 2024-03-11T21:04:39.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:39 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:04:39.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-116"}]': finished 2024-03-11T21:04:39.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:39 smithi129 ceph-mon[42066]: osdmap e591: 8 total, 8 up, 8 in 2024-03-11T21:04:39.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-116", "mode": "writeback"}]: dispatch 2024-03-11T21:04:39.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:39 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:04:39.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-116"}]': finished 2024-03-11T21:04:39.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:39 smithi157 ceph-mon[33359]: osdmap e591: 8 total, 8 up, 8 in 2024-03-11T21:04:39.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-116", "mode": "writeback"}]: dispatch 2024-03-11T21:04:39.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:39 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:04:40.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:40 smithi129 ceph-mon[42066]: pgmap v901: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T21:04:40.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-116", "mode": "writeback"}]': finished 2024-03-11T21:04:40.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:40 smithi129 ceph-mon[42066]: osdmap e592: 8 total, 8 up, 8 in 2024-03-11T21:04:40.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "325.7"}]: dispatch 2024-03-11T21:04:40.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:40 smithi129 ceph-mon[42066]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "325.7"}]: dispatch 2024-03-11T21:04:40.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:40 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:40.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:40 smithi129 ceph-mon[30184]: pgmap v901: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T21:04:40.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-116", "mode": "writeback"}]': finished 2024-03-11T21:04:40.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:40 smithi129 ceph-mon[30184]: osdmap e592: 8 total, 8 up, 8 in 2024-03-11T21:04:40.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "325.7"}]: dispatch 2024-03-11T21:04:40.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:40 smithi129 ceph-mon[30184]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "325.7"}]: dispatch 2024-03-11T21:04:40.978 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:40 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:40.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:40 smithi157 ceph-mon[33359]: pgmap v901: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-11T21:04:40.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-116", "mode": "writeback"}]': finished 2024-03-11T21:04:40.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:40 smithi157 ceph-mon[33359]: osdmap e592: 8 total, 8 up, 8 in 2024-03-11T21:04:40.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "325.7"}]: dispatch 2024-03-11T21:04:40.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:40 smithi157 ceph-mon[33359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "325.7"}]: dispatch 2024-03-11T21:04:40.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:40 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:04:41.976 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:41 smithi129 ceph-mon[30184]: 325.7 scrub starts 2024-03-11T21:04:41.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:41 smithi129 ceph-mon[30184]: 325.7 scrub ok 2024-03-11T21:04:41.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:41 smithi129 ceph-mon[42066]: 325.7 scrub starts 2024-03-11T21:04:41.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:41 smithi129 ceph-mon[42066]: 325.7 scrub ok 2024-03-11T21:04:41.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:41 smithi157 ceph-mon[33359]: 325.7 scrub starts 2024-03-11T21:04:41.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:41 smithi157 ceph-mon[33359]: 325.7 scrub ok 2024-03-11T21:04:42.885 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:42 smithi157 ceph-mon[33359]: pgmap v903: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-11T21:04:42.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:42 smithi129 ceph-mon[30184]: pgmap v903: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-11T21:04:42.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:42 smithi129 ceph-mon[42066]: pgmap v903: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-11T21:04:44.234 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:04:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:04:44.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:44 smithi129 ceph-mon[30184]: pgmap v904: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:44.978 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:44 smithi129 ceph-mon[42066]: pgmap v904: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:44.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:44 smithi157 ceph-mon[33359]: pgmap v904: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:45.976 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:45.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:45.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:46.476 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:04:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:04:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:04:46.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:46 smithi129 ceph-mon[30184]: pgmap v905: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 970 B/s rd, 1 op/s 2024-03-11T21:04:46.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:46 smithi129 ceph-mon[42066]: pgmap v905: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 970 B/s rd, 1 op/s 2024-03-11T21:04:46.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:46 smithi157 ceph-mon[33359]: pgmap v905: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 970 B/s rd, 1 op/s 2024-03-11T21:04:48.977 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:48 smithi129 ceph-mon[30184]: pgmap v906: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:04:48.977 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:48 smithi129 ceph-mon[42066]: pgmap v906: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:04:48.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:48 smithi157 ceph-mon[33359]: pgmap v906: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:04:50.226 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:49 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:04:50.227 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:49 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:50.227 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:49 smithi129 ceph-mon[30184]: pgmap v907: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:50.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:49 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:04:50.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:49 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:50.227 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:49 smithi129 ceph-mon[42066]: pgmap v907: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:50.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:49 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:04:50.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:49 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:04:50.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:49 smithi157 ceph-mon[33359]: pgmap v907: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:04:52.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:52 smithi129 ceph-mon[30184]: pgmap v908: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:04:52.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:52 smithi129 ceph-mon[42066]: pgmap v908: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:04:52.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:52 smithi157 ceph-mon[33359]: pgmap v908: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:04:53.194 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.194 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.194 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.194 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.194 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.194 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.194 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.194 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.194 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.194 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.195 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.195 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.195 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.195 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.195 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.195 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.195 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.195 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.195 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.195 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.195 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.195 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.195 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.196 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.196 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.196 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.196 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.196 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.196 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.196 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.196 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.196 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.196 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.196 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.196 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.196 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.196 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.197 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.197 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.197 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.197 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.197 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.197 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.197 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.197 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.198 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.198 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.198 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.198 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.198 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.198 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.198 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.198 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.198 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.198 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.198 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.198 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.198 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.198 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.199 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.199 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.199 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.199 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.199 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.199 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.199 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.199 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.199 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.200 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.200 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.200 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.200 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.200 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.200 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.200 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.200 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.200 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.200 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.200 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.200 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.200 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.200 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.201 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.201 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.201 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.201 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.201 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.201 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.201 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.201 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.201 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.201 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.201 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.201 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.201 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.202 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.202 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.202 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.202 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:04:53.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:53 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:04:53.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:53 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:04:53.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:53 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:04:54.234 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:04:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:04:54.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:54 smithi157 ceph-mon[33359]: pgmap v909: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:04:54.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:54 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:04:54.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:54 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:04:54.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:54 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:04:54.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:54 smithi129 ceph-mon[30184]: pgmap v909: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:04:54.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:54 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:04:54.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:54 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:04:54.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:54 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:04:54.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:54 smithi129 ceph-mon[42066]: pgmap v909: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:04:54.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:54 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:04:54.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:54 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:04:54.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:54 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:04:55.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:55.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:55.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:04:56.378 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:04:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:04:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:04:56.378 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:56 smithi129 ceph-mon[30184]: pgmap v910: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:04:56.378 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:56 smithi129 ceph-mon[30184]: osdmap e593: 8 total, 8 up, 8 in 2024-03-11T21:04:56.378 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:56.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:56 smithi129 ceph-mon[42066]: pgmap v910: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:04:56.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:56 smithi129 ceph-mon[42066]: osdmap e593: 8 total, 8 up, 8 in 2024-03-11T21:04:56.379 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:56.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:56 smithi157 ceph-mon[33359]: pgmap v910: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:04:56.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:56 smithi157 ceph-mon[33359]: osdmap e593: 8 total, 8 up, 8 in 2024-03-11T21:04:56.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:04:57.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:04:57.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:57 smithi129 ceph-mon[30184]: osdmap e594: 8 total, 8 up, 8 in 2024-03-11T21:04:57.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-116"}]: dispatch 2024-03-11T21:04:57.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:04:57.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:57 smithi129 ceph-mon[42066]: osdmap e594: 8 total, 8 up, 8 in 2024-03-11T21:04:57.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-116"}]: dispatch 2024-03-11T21:04:57.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:04:57.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:57 smithi157 ceph-mon[33359]: osdmap e594: 8 total, 8 up, 8 in 2024-03-11T21:04:57.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-116"}]: dispatch 2024-03-11T21:04:58.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:58 smithi129 ceph-mon[30184]: pgmap v913: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:04:58.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:58 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:04:58.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-116"}]': finished 2024-03-11T21:04:58.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:58 smithi129 ceph-mon[30184]: osdmap e595: 8 total, 8 up, 8 in 2024-03-11T21:04:58.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:58 smithi129 ceph-mon[42066]: pgmap v913: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:04:58.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:58 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:04:58.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-116"}]': finished 2024-03-11T21:04:58.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:58 smithi129 ceph-mon[42066]: osdmap e595: 8 total, 8 up, 8 in 2024-03-11T21:04:58.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:58 smithi157 ceph-mon[33359]: pgmap v913: 268 pgs: 268 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-11T21:04:58.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:58 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:04:58.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-116"}]': finished 2024-03-11T21:04:58.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:58 smithi157 ceph-mon[33359]: osdmap e595: 8 total, 8 up, 8 in 2024-03-11T21:04:59.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:59 smithi129 ceph-mon[30184]: osdmap e596: 8 total, 8 up, 8 in 2024-03-11T21:04:59.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:59 smithi129 ceph-mon[30184]: osdmap e597: 8 total, 8 up, 8 in 2024-03-11T21:04:59.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:04:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:04:59.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:59 smithi129 ceph-mon[42066]: osdmap e596: 8 total, 8 up, 8 in 2024-03-11T21:04:59.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:59 smithi129 ceph-mon[42066]: osdmap e597: 8 total, 8 up, 8 in 2024-03-11T21:04:59.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:04:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:04:59.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:59 smithi157 ceph-mon[33359]: osdmap e596: 8 total, 8 up, 8 in 2024-03-11T21:04:59.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:59 smithi157 ceph-mon[33359]: osdmap e597: 8 total, 8 up, 8 in 2024-03-11T21:04:59.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:04:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:00.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:00 smithi129 ceph-mon[30184]: pgmap v916: 236 pgs: 236 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-11T21:05:00.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:00 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:00.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:00 smithi129 ceph-mon[30184]: osdmap e598: 8 total, 8 up, 8 in 2024-03-11T21:05:00.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:00 smithi129 ceph-mon[42066]: pgmap v916: 236 pgs: 236 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-11T21:05:00.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:00 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:00.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:00 smithi129 ceph-mon[42066]: osdmap e598: 8 total, 8 up, 8 in 2024-03-11T21:05:00.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:00 smithi157 ceph-mon[33359]: pgmap v916: 236 pgs: 236 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-11T21:05:00.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:00 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:00.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:00 smithi157 ceph-mon[33359]: osdmap e598: 8 total, 8 up, 8 in 2024-03-11T21:05:02.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:02 smithi129 ceph-mon[30184]: pgmap v919: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:02.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:02 smithi129 ceph-mon[30184]: osdmap e599: 8 total, 8 up, 8 in 2024-03-11T21:05:02.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:02 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:02.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:02 smithi129 ceph-mon[42066]: pgmap v919: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:02.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:02 smithi129 ceph-mon[42066]: osdmap e599: 8 total, 8 up, 8 in 2024-03-11T21:05:02.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:02 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:02.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:02 smithi157 ceph-mon[33359]: pgmap v919: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:02.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:02 smithi157 ceph-mon[33359]: osdmap e599: 8 total, 8 up, 8 in 2024-03-11T21:05:02.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:02 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:03.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:03.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:03 smithi129 ceph-mon[30184]: osdmap e600: 8 total, 8 up, 8 in 2024-03-11T21:05:03.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-118"}]: dispatch 2024-03-11T21:05:03.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:03.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:03 smithi129 ceph-mon[42066]: osdmap e600: 8 total, 8 up, 8 in 2024-03-11T21:05:03.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-118"}]: dispatch 2024-03-11T21:05:03.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:03.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:03 smithi157 ceph-mon[33359]: osdmap e600: 8 total, 8 up, 8 in 2024-03-11T21:05:03.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-118"}]: dispatch 2024-03-11T21:05:04.234 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:05:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:05:04.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:04 smithi157 ceph-mon[33359]: pgmap v922: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:05:04.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-118"}]': finished 2024-03-11T21:05:04.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:04 smithi157 ceph-mon[33359]: osdmap e601: 8 total, 8 up, 8 in 2024-03-11T21:05:04.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-118", "mode": "writeback"}]: dispatch 2024-03-11T21:05:04.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:05:04.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:04.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:04 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:04.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-118", "mode": "writeback"}]': finished 2024-03-11T21:05:04.235 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:04 smithi157 ceph-mon[33359]: osdmap e602: 8 total, 8 up, 8 in 2024-03-11T21:05:04.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[30184]: pgmap v922: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:05:04.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-118"}]': finished 2024-03-11T21:05:04.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[30184]: osdmap e601: 8 total, 8 up, 8 in 2024-03-11T21:05:04.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-118", "mode": "writeback"}]: dispatch 2024-03-11T21:05:04.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:05:04.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:04.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:04.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-118", "mode": "writeback"}]': finished 2024-03-11T21:05:04.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[30184]: osdmap e602: 8 total, 8 up, 8 in 2024-03-11T21:05:04.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[42066]: pgmap v922: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:05:04.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-118"}]': finished 2024-03-11T21:05:04.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[42066]: osdmap e601: 8 total, 8 up, 8 in 2024-03-11T21:05:04.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-118", "mode": "writeback"}]: dispatch 2024-03-11T21:05:04.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:05:04.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:04.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:04.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-118", "mode": "writeback"}]': finished 2024-03-11T21:05:04.478 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:04 smithi129 ceph-mon[42066]: osdmap e602: 8 total, 8 up, 8 in 2024-03-11T21:05:05.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:05.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:05 smithi129 ceph-mon[30184]: osdmap e603: 8 total, 8 up, 8 in 2024-03-11T21:05:05.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:05.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:05 smithi129 ceph-mon[42066]: osdmap e603: 8 total, 8 up, 8 in 2024-03-11T21:05:05.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:05.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:05 smithi157 ceph-mon[33359]: osdmap e603: 8 total, 8 up, 8 in 2024-03-11T21:05:06.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:06 smithi129 ceph-mon[30184]: pgmap v925: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:05:06.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:06.477 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:05:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:05:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:05:06.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:06 smithi129 ceph-mon[42066]: pgmap v925: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:05:06.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:06.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:06 smithi157 ceph-mon[33359]: pgmap v925: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:05:06.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:07.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:07.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:07 smithi129 ceph-mon[30184]: osdmap e604: 8 total, 8 up, 8 in 2024-03-11T21:05:07.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-118"}]: dispatch 2024-03-11T21:05:07.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:07.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:07 smithi129 ceph-mon[42066]: osdmap e604: 8 total, 8 up, 8 in 2024-03-11T21:05:07.479 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-118"}]: dispatch 2024-03-11T21:05:07.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:07.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:07 smithi157 ceph-mon[33359]: osdmap e604: 8 total, 8 up, 8 in 2024-03-11T21:05:07.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-118"}]: dispatch 2024-03-11T21:05:08.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:08 smithi129 ceph-mon[30184]: pgmap v928: 268 pgs: 268 active+clean; 582 KiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:05:08.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:08 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:08.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-118"}]': finished 2024-03-11T21:05:08.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:08 smithi129 ceph-mon[30184]: osdmap e605: 8 total, 8 up, 8 in 2024-03-11T21:05:08.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:08 smithi129 ceph-mon[42066]: pgmap v928: 268 pgs: 268 active+clean; 582 KiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:05:08.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:08 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:08.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-118"}]': finished 2024-03-11T21:05:08.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:08 smithi129 ceph-mon[42066]: osdmap e605: 8 total, 8 up, 8 in 2024-03-11T21:05:08.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:08 smithi157 ceph-mon[33359]: pgmap v928: 268 pgs: 268 active+clean; 582 KiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:05:08.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:08 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:08.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-118"}]': finished 2024-03-11T21:05:08.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:08 smithi157 ceph-mon[33359]: osdmap e605: 8 total, 8 up, 8 in 2024-03-11T21:05:09.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:09 smithi129 ceph-mon[30184]: osdmap e606: 8 total, 8 up, 8 in 2024-03-11T21:05:09.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:09 smithi129 ceph-mon[30184]: osdmap e607: 8 total, 8 up, 8 in 2024-03-11T21:05:09.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:09.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:09 smithi129 ceph-mon[42066]: osdmap e606: 8 total, 8 up, 8 in 2024-03-11T21:05:09.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:09 smithi129 ceph-mon[42066]: osdmap e607: 8 total, 8 up, 8 in 2024-03-11T21:05:09.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:09.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:09 smithi157 ceph-mon[33359]: osdmap e606: 8 total, 8 up, 8 in 2024-03-11T21:05:09.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:09 smithi157 ceph-mon[33359]: osdmap e607: 8 total, 8 up, 8 in 2024-03-11T21:05:09.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:10.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:10 smithi129 ceph-mon[30184]: pgmap v931: 236 pgs: 236 active+clean; 582 KiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:05:10.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:10.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:10 smithi129 ceph-mon[30184]: osdmap e608: 8 total, 8 up, 8 in 2024-03-11T21:05:10.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:10.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:10 smithi129 ceph-mon[42066]: pgmap v931: 236 pgs: 236 active+clean; 582 KiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:05:10.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:10.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:10 smithi129 ceph-mon[42066]: osdmap e608: 8 total, 8 up, 8 in 2024-03-11T21:05:10.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:10.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:10 smithi157 ceph-mon[33359]: pgmap v931: 236 pgs: 236 active+clean; 582 KiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:05:10.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:10.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:10 smithi157 ceph-mon[33359]: osdmap e608: 8 total, 8 up, 8 in 2024-03-11T21:05:10.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:12.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:12 smithi129 ceph-mon[30184]: pgmap v934: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:12.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:12.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:12 smithi129 ceph-mon[30184]: osdmap e609: 8 total, 8 up, 8 in 2024-03-11T21:05:12.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-120"}]: dispatch 2024-03-11T21:05:12.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:12 smithi129 ceph-mon[42066]: pgmap v934: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:12.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:12.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:12 smithi129 ceph-mon[42066]: osdmap e609: 8 total, 8 up, 8 in 2024-03-11T21:05:12.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-120"}]: dispatch 2024-03-11T21:05:12.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:12 smithi157 ceph-mon[33359]: pgmap v934: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:12.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:12.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:12 smithi157 ceph-mon[33359]: osdmap e609: 8 total, 8 up, 8 in 2024-03-11T21:05:12.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-120"}]: dispatch 2024-03-11T21:05:13.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-120"}]': finished 2024-03-11T21:05:13.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:13 smithi129 ceph-mon[42066]: osdmap e610: 8 total, 8 up, 8 in 2024-03-11T21:05:13.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-120", "mode": "writeback"}]: dispatch 2024-03-11T21:05:13.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:13 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:13.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-120", "mode": "writeback"}]': finished 2024-03-11T21:05:13.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:13 smithi129 ceph-mon[42066]: osdmap e611: 8 total, 8 up, 8 in 2024-03-11T21:05:13.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-120"}]': finished 2024-03-11T21:05:13.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:13 smithi129 ceph-mon[30184]: osdmap e610: 8 total, 8 up, 8 in 2024-03-11T21:05:13.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-120", "mode": "writeback"}]: dispatch 2024-03-11T21:05:13.478 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:13 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:13.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-120", "mode": "writeback"}]': finished 2024-03-11T21:05:13.479 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:13 smithi129 ceph-mon[30184]: osdmap e611: 8 total, 8 up, 8 in 2024-03-11T21:05:13.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-120"}]': finished 2024-03-11T21:05:13.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:13 smithi157 ceph-mon[33359]: osdmap e610: 8 total, 8 up, 8 in 2024-03-11T21:05:13.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-120", "mode": "writeback"}]: dispatch 2024-03-11T21:05:13.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:13 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:13.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-120", "mode": "writeback"}]': finished 2024-03-11T21:05:13.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:13 smithi157 ceph-mon[33359]: osdmap e611: 8 total, 8 up, 8 in 2024-03-11T21:05:13.983 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:05:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:05:14.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:14 smithi129 ceph-mon[30184]: pgmap v937: 268 pgs: 268 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:14.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:14.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:14 smithi129 ceph-mon[42066]: pgmap v937: 268 pgs: 268 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:14.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:14.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:14 smithi157 ceph-mon[33359]: pgmap v937: 268 pgs: 268 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:14.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:15.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:15.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:15.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:15 smithi129 ceph-mon[30184]: osdmap e612: 8 total, 8 up, 8 in 2024-03-11T21:05:15.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-120"}]: dispatch 2024-03-11T21:05:15.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:15.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:15.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:15 smithi129 ceph-mon[42066]: osdmap e612: 8 total, 8 up, 8 in 2024-03-11T21:05:15.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-120"}]: dispatch 2024-03-11T21:05:15.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:15.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:15.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:15 smithi157 ceph-mon[33359]: osdmap e612: 8 total, 8 up, 8 in 2024-03-11T21:05:15.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-120"}]: dispatch 2024-03-11T21:05:16.476 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:05:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:05:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:05:16.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:16 smithi129 ceph-mon[30184]: pgmap v940: 268 pgs: 268 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:16.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:16 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:16.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-120"}]': finished 2024-03-11T21:05:16.477 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:16 smithi129 ceph-mon[30184]: osdmap e613: 8 total, 8 up, 8 in 2024-03-11T21:05:16.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:16 smithi129 ceph-mon[42066]: pgmap v940: 268 pgs: 268 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:16.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:16 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:16.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-120"}]': finished 2024-03-11T21:05:16.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:16 smithi129 ceph-mon[42066]: osdmap e613: 8 total, 8 up, 8 in 2024-03-11T21:05:16.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:16 smithi157 ceph-mon[33359]: pgmap v940: 268 pgs: 268 active+clean; 582 KiB data, 642 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:16.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:16 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:16.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-120"}]': finished 2024-03-11T21:05:16.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:16 smithi157 ceph-mon[33359]: osdmap e613: 8 total, 8 up, 8 in 2024-03-11T21:05:17.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:17 smithi129 ceph-mon[30184]: osdmap e614: 8 total, 8 up, 8 in 2024-03-11T21:05:17.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:17 smithi129 ceph-mon[30184]: osdmap e615: 8 total, 8 up, 8 in 2024-03-11T21:05:17.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:17 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:17.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:17 smithi129 ceph-mon[42066]: osdmap e614: 8 total, 8 up, 8 in 2024-03-11T21:05:17.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:17 smithi129 ceph-mon[42066]: osdmap e615: 8 total, 8 up, 8 in 2024-03-11T21:05:17.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:17 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:17.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:17 smithi157 ceph-mon[33359]: osdmap e614: 8 total, 8 up, 8 in 2024-03-11T21:05:17.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:17 smithi157 ceph-mon[33359]: osdmap e615: 8 total, 8 up, 8 in 2024-03-11T21:05:17.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:17 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:18.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:18 smithi129 ceph-mon[30184]: pgmap v943: 236 pgs: 236 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:18.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:18.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:18 smithi129 ceph-mon[30184]: osdmap e616: 8 total, 8 up, 8 in 2024-03-11T21:05:18.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:18.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:18 smithi129 ceph-mon[42066]: pgmap v943: 236 pgs: 236 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:18.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:18.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:18 smithi129 ceph-mon[42066]: osdmap e616: 8 total, 8 up, 8 in 2024-03-11T21:05:18.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:18.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:18 smithi157 ceph-mon[33359]: pgmap v943: 236 pgs: 236 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:18.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:18.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:18 smithi157 ceph-mon[33359]: osdmap e616: 8 total, 8 up, 8 in 2024-03-11T21:05:18.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:19.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:19.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:19.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:19 smithi129 ceph-mon[30184]: osdmap e617: 8 total, 8 up, 8 in 2024-03-11T21:05:19.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-122"}]: dispatch 2024-03-11T21:05:19.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:19.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:19.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:19 smithi129 ceph-mon[42066]: osdmap e617: 8 total, 8 up, 8 in 2024-03-11T21:05:19.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-122"}]: dispatch 2024-03-11T21:05:19.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:19.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:19.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:19 smithi157 ceph-mon[33359]: osdmap e617: 8 total, 8 up, 8 in 2024-03-11T21:05:19.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-122"}]: dispatch 2024-03-11T21:05:20.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:20 smithi129 ceph-mon[30184]: pgmap v946: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:20.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-122"}]': finished 2024-03-11T21:05:20.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:20 smithi129 ceph-mon[30184]: osdmap e618: 8 total, 8 up, 8 in 2024-03-11T21:05:20.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-122", "mode": "writeback"}]: dispatch 2024-03-11T21:05:20.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:20 smithi129 ceph-mon[42066]: pgmap v946: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:20.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-122"}]': finished 2024-03-11T21:05:20.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:20 smithi129 ceph-mon[42066]: osdmap e618: 8 total, 8 up, 8 in 2024-03-11T21:05:20.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-122", "mode": "writeback"}]: dispatch 2024-03-11T21:05:20.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:20 smithi157 ceph-mon[33359]: pgmap v946: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:20.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-122"}]': finished 2024-03-11T21:05:20.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:20 smithi157 ceph-mon[33359]: osdmap e618: 8 total, 8 up, 8 in 2024-03-11T21:05:20.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-122", "mode": "writeback"}]: dispatch 2024-03-11T21:05:21.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:21 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:21.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-122", "mode": "writeback"}]': finished 2024-03-11T21:05:21.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:21 smithi129 ceph-mon[30184]: osdmap e619: 8 total, 8 up, 8 in 2024-03-11T21:05:21.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:21.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:21 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:21.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-122", "mode": "writeback"}]': finished 2024-03-11T21:05:21.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:21 smithi129 ceph-mon[42066]: osdmap e619: 8 total, 8 up, 8 in 2024-03-11T21:05:21.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:21.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:21 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:21.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-122", "mode": "writeback"}]': finished 2024-03-11T21:05:21.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:21 smithi157 ceph-mon[33359]: osdmap e619: 8 total, 8 up, 8 in 2024-03-11T21:05:21.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:22.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:22 smithi129 ceph-mon[30184]: pgmap v949: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:22.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:22.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:22 smithi129 ceph-mon[30184]: osdmap e620: 8 total, 8 up, 8 in 2024-03-11T21:05:22.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-122"}]: dispatch 2024-03-11T21:05:22.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:22 smithi129 ceph-mon[42066]: pgmap v949: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:22.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:22.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:22 smithi129 ceph-mon[42066]: osdmap e620: 8 total, 8 up, 8 in 2024-03-11T21:05:22.727 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-122"}]: dispatch 2024-03-11T21:05:22.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:22 smithi157 ceph-mon[33359]: pgmap v949: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:22.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:22.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:22 smithi157 ceph-mon[33359]: osdmap e620: 8 total, 8 up, 8 in 2024-03-11T21:05:22.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-122"}]: dispatch 2024-03-11T21:05:23.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:23 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:23.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-122"}]': finished 2024-03-11T21:05:23.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:23 smithi157 ceph-mon[33359]: osdmap e621: 8 total, 8 up, 8 in 2024-03-11T21:05:23.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:23 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:23.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-122"}]': finished 2024-03-11T21:05:23.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:23 smithi129 ceph-mon[30184]: osdmap e621: 8 total, 8 up, 8 in 2024-03-11T21:05:23.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:23 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:23.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-122"}]': finished 2024-03-11T21:05:23.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:23 smithi129 ceph-mon[42066]: osdmap e621: 8 total, 8 up, 8 in 2024-03-11T21:05:24.233 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:05:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:05:24.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:24 smithi129 ceph-mon[30184]: pgmap v952: 268 pgs: 268 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-11T21:05:24.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:24 smithi129 ceph-mon[30184]: osdmap e622: 8 total, 8 up, 8 in 2024-03-11T21:05:24.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:24 smithi129 ceph-mon[42066]: pgmap v952: 268 pgs: 268 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-11T21:05:24.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:24 smithi129 ceph-mon[42066]: osdmap e622: 8 total, 8 up, 8 in 2024-03-11T21:05:24.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:24 smithi157 ceph-mon[33359]: pgmap v952: 268 pgs: 268 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-11T21:05:24.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:24 smithi157 ceph-mon[33359]: osdmap e622: 8 total, 8 up, 8 in 2024-03-11T21:05:25.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:25.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:25 smithi129 ceph-mon[42066]: osdmap e623: 8 total, 8 up, 8 in 2024-03-11T21:05:25.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:25.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:25.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:25 smithi129 ceph-mon[30184]: osdmap e623: 8 total, 8 up, 8 in 2024-03-11T21:05:25.727 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:25.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:25.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:25 smithi157 ceph-mon[33359]: osdmap e623: 8 total, 8 up, 8 in 2024-03-11T21:05:25.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:26.355 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:05:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:05:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:05:26.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:26 smithi129 ceph-mon[30184]: pgmap v955: 236 pgs: 236 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:05:26.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:26 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:26.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:26 smithi129 ceph-mon[30184]: osdmap e624: 8 total, 8 up, 8 in 2024-03-11T21:05:26.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:26 smithi129 ceph-mon[42066]: pgmap v955: 236 pgs: 236 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:05:26.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:26 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:26.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:26 smithi129 ceph-mon[42066]: osdmap e624: 8 total, 8 up, 8 in 2024-03-11T21:05:26.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:26 smithi157 ceph-mon[33359]: pgmap v955: 236 pgs: 236 active+clean; 582 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-11T21:05:26.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:26 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:26.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:26 smithi157 ceph-mon[33359]: osdmap e624: 8 total, 8 up, 8 in 2024-03-11T21:05:28.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:28 smithi129 ceph-mon[30184]: pgmap v958: 268 pgs: 268 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:05:28.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:28 smithi129 ceph-mon[30184]: osdmap e625: 8 total, 8 up, 8 in 2024-03-11T21:05:28.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:28.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:28 smithi129 ceph-mon[42066]: pgmap v958: 268 pgs: 268 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:05:28.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:28 smithi129 ceph-mon[42066]: osdmap e625: 8 total, 8 up, 8 in 2024-03-11T21:05:28.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:28.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:28 smithi157 ceph-mon[33359]: pgmap v958: 268 pgs: 268 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:05:28.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:28 smithi157 ceph-mon[33359]: osdmap e625: 8 total, 8 up, 8 in 2024-03-11T21:05:28.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:29.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:29.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:29 smithi129 ceph-mon[30184]: osdmap e626: 8 total, 8 up, 8 in 2024-03-11T21:05:29.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-124"}]: dispatch 2024-03-11T21:05:29.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:29.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:29 smithi129 ceph-mon[42066]: osdmap e626: 8 total, 8 up, 8 in 2024-03-11T21:05:29.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-124"}]: dispatch 2024-03-11T21:05:29.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:29.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:29 smithi157 ceph-mon[33359]: osdmap e626: 8 total, 8 up, 8 in 2024-03-11T21:05:29.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-124"}]: dispatch 2024-03-11T21:05:30.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:30 smithi129 ceph-mon[30184]: pgmap v961: 268 pgs: 268 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:05:30.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-124"}]': finished 2024-03-11T21:05:30.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:30 smithi129 ceph-mon[30184]: osdmap e627: 8 total, 8 up, 8 in 2024-03-11T21:05:30.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-124", "mode": "writeback"}]: dispatch 2024-03-11T21:05:30.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:30 smithi129 ceph-mon[42066]: pgmap v961: 268 pgs: 268 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:05:30.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-124"}]': finished 2024-03-11T21:05:30.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:30 smithi129 ceph-mon[42066]: osdmap e627: 8 total, 8 up, 8 in 2024-03-11T21:05:30.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-124", "mode": "writeback"}]: dispatch 2024-03-11T21:05:30.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:30 smithi157 ceph-mon[33359]: pgmap v961: 268 pgs: 268 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:05:30.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-124"}]': finished 2024-03-11T21:05:30.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:30 smithi157 ceph-mon[33359]: osdmap e627: 8 total, 8 up, 8 in 2024-03-11T21:05:30.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-124", "mode": "writeback"}]: dispatch 2024-03-11T21:05:31.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:31 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:31.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-124", "mode": "writeback"}]': finished 2024-03-11T21:05:31.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:31 smithi129 ceph-mon[30184]: osdmap e628: 8 total, 8 up, 8 in 2024-03-11T21:05:31.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:31 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:31.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-124", "mode": "writeback"}]': finished 2024-03-11T21:05:31.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:31 smithi129 ceph-mon[42066]: osdmap e628: 8 total, 8 up, 8 in 2024-03-11T21:05:31.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:31 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:31.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-124", "mode": "writeback"}]': finished 2024-03-11T21:05:31.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:31 smithi157 ceph-mon[33359]: osdmap e628: 8 total, 8 up, 8 in 2024-03-11T21:05:32.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:32 smithi129 ceph-mon[42066]: pgmap v964: 268 pgs: 268 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:32.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:32 smithi129 ceph-mon[42066]: osdmap e629: 8 total, 8 up, 8 in 2024-03-11T21:05:32.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:32 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:32.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:32 smithi129 ceph-mon[30184]: pgmap v964: 268 pgs: 268 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:32.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:32 smithi129 ceph-mon[30184]: osdmap e629: 8 total, 8 up, 8 in 2024-03-11T21:05:32.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:32 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:32.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:32 smithi157 ceph-mon[33359]: pgmap v964: 268 pgs: 268 active+clean; 582 KiB data, 681 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:32.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:32 smithi157 ceph-mon[33359]: osdmap e629: 8 total, 8 up, 8 in 2024-03-11T21:05:32.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:32 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:33.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:33.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:33 smithi129 ceph-mon[30184]: osdmap e630: 8 total, 8 up, 8 in 2024-03-11T21:05:33.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-124"}]: dispatch 2024-03-11T21:05:33.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:33 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:33.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-124"}]': finished 2024-03-11T21:05:33.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:33 smithi129 ceph-mon[30184]: osdmap e631: 8 total, 8 up, 8 in 2024-03-11T21:05:33.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:33.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:33 smithi129 ceph-mon[42066]: osdmap e630: 8 total, 8 up, 8 in 2024-03-11T21:05:33.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-124"}]: dispatch 2024-03-11T21:05:33.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:33 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:33.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-124"}]': finished 2024-03-11T21:05:33.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:33 smithi129 ceph-mon[42066]: osdmap e631: 8 total, 8 up, 8 in 2024-03-11T21:05:33.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:33.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:33 smithi157 ceph-mon[33359]: osdmap e630: 8 total, 8 up, 8 in 2024-03-11T21:05:33.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-124"}]: dispatch 2024-03-11T21:05:33.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:33 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:33.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-124"}]': finished 2024-03-11T21:05:33.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:33 smithi157 ceph-mon[33359]: osdmap e631: 8 total, 8 up, 8 in 2024-03-11T21:05:34.233 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:05:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:05:34.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:34 smithi129 ceph-mon[30184]: pgmap v967: 268 pgs: 268 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-11T21:05:34.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:34 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:34.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:34 smithi129 ceph-mon[42066]: pgmap v967: 268 pgs: 268 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-11T21:05:34.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:34 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:34.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:34 smithi157 ceph-mon[33359]: pgmap v967: 268 pgs: 268 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-11T21:05:34.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:34 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:35.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:35.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:35 smithi129 ceph-mon[30184]: osdmap e632: 8 total, 8 up, 8 in 2024-03-11T21:05:35.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:35 smithi129 ceph-mon[30184]: osdmap e633: 8 total, 8 up, 8 in 2024-03-11T21:05:35.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:35.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:35.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:35 smithi129 ceph-mon[42066]: osdmap e632: 8 total, 8 up, 8 in 2024-03-11T21:05:35.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:35 smithi129 ceph-mon[42066]: osdmap e633: 8 total, 8 up, 8 in 2024-03-11T21:05:35.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:35.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:35.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:35 smithi157 ceph-mon[33359]: osdmap e632: 8 total, 8 up, 8 in 2024-03-11T21:05:35.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:35 smithi157 ceph-mon[33359]: osdmap e633: 8 total, 8 up, 8 in 2024-03-11T21:05:35.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:36.392 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:05:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:05:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:05:36.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:36 smithi129 ceph-mon[30184]: pgmap v970: 236 pgs: 236 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T21:05:36.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:36.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:36 smithi129 ceph-mon[30184]: osdmap e634: 8 total, 8 up, 8 in 2024-03-11T21:05:36.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:36.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:36 smithi129 ceph-mon[42066]: pgmap v970: 236 pgs: 236 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T21:05:36.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:36.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:36 smithi129 ceph-mon[42066]: osdmap e634: 8 total, 8 up, 8 in 2024-03-11T21:05:36.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:36.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:36 smithi157 ceph-mon[33359]: pgmap v970: 236 pgs: 236 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-11T21:05:36.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:36.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:36 smithi157 ceph-mon[33359]: osdmap e634: 8 total, 8 up, 8 in 2024-03-11T21:05:36.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:38.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:38 smithi129 ceph-mon[30184]: pgmap v973: 268 pgs: 268 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:38.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:38.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:38 smithi129 ceph-mon[30184]: osdmap e635: 8 total, 8 up, 8 in 2024-03-11T21:05:38.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-126"}]: dispatch 2024-03-11T21:05:38.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:38 smithi129 ceph-mon[42066]: pgmap v973: 268 pgs: 268 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:38.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:38.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:38 smithi129 ceph-mon[42066]: osdmap e635: 8 total, 8 up, 8 in 2024-03-11T21:05:38.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-126"}]: dispatch 2024-03-11T21:05:38.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:38 smithi157 ceph-mon[33359]: pgmap v973: 268 pgs: 268 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:38.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:38.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:38 smithi157 ceph-mon[33359]: osdmap e635: 8 total, 8 up, 8 in 2024-03-11T21:05:38.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-126"}]: dispatch 2024-03-11T21:05:39.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-126"}]': finished 2024-03-11T21:05:39.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:39 smithi129 ceph-mon[30184]: osdmap e636: 8 total, 8 up, 8 in 2024-03-11T21:05:39.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-126", "mode": "writeback"}]: dispatch 2024-03-11T21:05:39.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:39 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:39.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-126", "mode": "writeback"}]': finished 2024-03-11T21:05:39.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:39 smithi129 ceph-mon[30184]: osdmap e637: 8 total, 8 up, 8 in 2024-03-11T21:05:39.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-126"}]': finished 2024-03-11T21:05:39.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:39 smithi129 ceph-mon[42066]: osdmap e636: 8 total, 8 up, 8 in 2024-03-11T21:05:39.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-126", "mode": "writeback"}]: dispatch 2024-03-11T21:05:39.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:39 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:39.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-126", "mode": "writeback"}]': finished 2024-03-11T21:05:39.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:39 smithi129 ceph-mon[42066]: osdmap e637: 8 total, 8 up, 8 in 2024-03-11T21:05:39.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-126"}]': finished 2024-03-11T21:05:39.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:39 smithi157 ceph-mon[33359]: osdmap e636: 8 total, 8 up, 8 in 2024-03-11T21:05:39.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-126", "mode": "writeback"}]: dispatch 2024-03-11T21:05:39.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:39 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:39.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-126", "mode": "writeback"}]': finished 2024-03-11T21:05:39.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:39 smithi157 ceph-mon[33359]: osdmap e637: 8 total, 8 up, 8 in 2024-03-11T21:05:40.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:40 smithi129 ceph-mon[30184]: pgmap v976: 268 pgs: 268 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:05:40.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:40.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:40 smithi129 ceph-mon[42066]: pgmap v976: 268 pgs: 268 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:05:40.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:40.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:40 smithi157 ceph-mon[33359]: pgmap v976: 268 pgs: 268 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:05:40.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:41.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:41 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:41.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:41 smithi129 ceph-mon[30184]: osdmap e638: 8 total, 8 up, 8 in 2024-03-11T21:05:41.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:41 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-126"}]: dispatch 2024-03-11T21:05:41.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:41 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:41.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:41 smithi129 ceph-mon[42066]: osdmap e638: 8 total, 8 up, 8 in 2024-03-11T21:05:41.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:41 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-126"}]: dispatch 2024-03-11T21:05:41.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:41 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:41.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:41 smithi157 ceph-mon[33359]: osdmap e638: 8 total, 8 up, 8 in 2024-03-11T21:05:41.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:41 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-126"}]: dispatch 2024-03-11T21:05:42.088 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 seapi_tier_pp: require_osd_release = squid 2024-03-11T21:05:42.088 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9062 ms) 2024-03-11T21:05:42.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-03-11T21:05:42.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:05:42.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6084 ms) 2024-03-11T21:05:42.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-03-11T21:05:42.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:05:42.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5041 ms) 2024-03-11T21:05:42.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-03-11T21:05:42.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:05:42.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26081 ms) 2024-03-11T21:05:42.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-03-11T21:05:42.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:05:42.089 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13260 ms) 2024-03-11T21:05:42.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-03-11T21:05:42.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12066 ms) 2024-03-11T21:05:42.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-03-11T21:05:42.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-03-11T21:05:42.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 00000010 69 73 |is| 2024-03-11T21:05:42.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 00000012 2024-03-11T21:05:42.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12076 ms) 2024-03-11T21:05:42.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-03-11T21:05:42.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: require_osd_release = squid 2024-03-11T21:05:42.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6052 ms) 2024-03-11T21:05:42.090 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (555821 ms total) 2024-03-11T21:05:42.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 2024-03-11T21:05:42.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-03-11T21:05:42.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-03-11T21:05:42.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1016 ms) 2024-03-11T21:05:42.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-03-11T21:05:42.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11033 ms) 2024-03-11T21:05:42.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-03-11T21:05:42.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18127 ms) 2024-03-11T21:05:42.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-03-11T21:05:42.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-03-11T21:05:42.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30178 ms total) 2024-03-11T21:05:42.091 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 2024-03-11T21:05:42.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-03-11T21:05:42.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-03-11T21:05:42.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7061 ms) 2024-03-11T21:05:42.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-03-11T21:05:42.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8045 ms) 2024-03-11T21:05:42.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-03-11T21:05:42.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: waiting for scrub... 2024-03-11T21:05:42.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: done waiting 2024-03-11T21:05:42.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24487 ms) 2024-03-11T21:05:42.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-03-11T21:05:42.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10034 ms) 2024-03-11T21:05:42.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-03-11T21:05:42.092 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7987 ms) 2024-03-11T21:05:42.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-03-11T21:05:42.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8013 ms) 2024-03-11T21:05:42.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-03-11T21:05:42.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10053 ms) 2024-03-11T21:05:42.093 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-03-11T21:05:42.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:42 smithi129 ceph-mon[30184]: pgmap v979: 268 pgs: 268 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:42.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:42 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:42.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-126"}]': finished 2024-03-11T21:05:42.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:42 smithi129 ceph-mon[30184]: osdmap e639: 8 total, 8 up, 8 in 2024-03-11T21:05:42.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:42 smithi129 ceph-mon[42066]: pgmap v979: 268 pgs: 268 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:42.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:42 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:42.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-126"}]': finished 2024-03-11T21:05:42.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:42 smithi129 ceph-mon[42066]: osdmap e639: 8 total, 8 up, 8 in 2024-03-11T21:05:42.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:42 smithi157 ceph-mon[33359]: pgmap v979: 268 pgs: 268 active+clean; 582 KiB data, 682 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:05:42.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:42 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:42.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-126"}]': finished 2024-03-11T21:05:42.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:42 smithi157 ceph-mon[33359]: osdmap e639: 8 total, 8 up, 8 in 2024-03-11T21:05:43.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:43 smithi129 ceph-mon[30184]: osdmap e640: 8 total, 8 up, 8 in 2024-03-11T21:05:43.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:43 smithi129 ceph-mon[30184]: osdmap e641: 8 total, 8 up, 8 in 2024-03-11T21:05:43.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:43.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:43 smithi129 ceph-mon[42066]: osdmap e640: 8 total, 8 up, 8 in 2024-03-11T21:05:43.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:43 smithi129 ceph-mon[42066]: osdmap e641: 8 total, 8 up, 8 in 2024-03-11T21:05:43.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:43.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:43 smithi157 ceph-mon[33359]: osdmap e640: 8 total, 8 up, 8 in 2024-03-11T21:05:43.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:43 smithi157 ceph-mon[33359]: osdmap e641: 8 total, 8 up, 8 in 2024-03-11T21:05:43.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:44.233 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:05:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:05:44.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:44 smithi129 ceph-mon[30184]: pgmap v982: 236 pgs: 236 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:44.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:44.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:44 smithi129 ceph-mon[30184]: osdmap e642: 8 total, 8 up, 8 in 2024-03-11T21:05:44.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:44.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:44 smithi129 ceph-mon[42066]: pgmap v982: 236 pgs: 236 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:44.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:44.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:44 smithi129 ceph-mon[42066]: osdmap e642: 8 total, 8 up, 8 in 2024-03-11T21:05:44.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:44.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:44 smithi157 ceph-mon[33359]: pgmap v982: 236 pgs: 236 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:44.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:44.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:44 smithi157 ceph-mon[33359]: osdmap e642: 8 total, 8 up, 8 in 2024-03-11T21:05:44.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:45.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:45.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:45.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:45 smithi129 ceph-mon[30184]: osdmap e643: 8 total, 8 up, 8 in 2024-03-11T21:05:45.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-128"}]: dispatch 2024-03-11T21:05:45.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:45.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:45.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:45 smithi129 ceph-mon[42066]: osdmap e643: 8 total, 8 up, 8 in 2024-03-11T21:05:45.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-128"}]: dispatch 2024-03-11T21:05:45.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:45.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:45.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:45 smithi157 ceph-mon[33359]: osdmap e643: 8 total, 8 up, 8 in 2024-03-11T21:05:45.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-128"}]: dispatch 2024-03-11T21:05:46.475 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:05:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:05:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:05:46.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:46 smithi129 ceph-mon[30184]: pgmap v985: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:46.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-128"}]': finished 2024-03-11T21:05:46.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:46 smithi129 ceph-mon[30184]: osdmap e644: 8 total, 8 up, 8 in 2024-03-11T21:05:46.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-128", "mode": "writeback"}]: dispatch 2024-03-11T21:05:46.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:46 smithi129 ceph-mon[42066]: pgmap v985: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:46.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:46 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-128"}]': finished 2024-03-11T21:05:46.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:46 smithi129 ceph-mon[42066]: osdmap e644: 8 total, 8 up, 8 in 2024-03-11T21:05:46.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:46 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-128", "mode": "writeback"}]: dispatch 2024-03-11T21:05:46.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:46 smithi157 ceph-mon[33359]: pgmap v985: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:05:46.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:46 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-128"}]': finished 2024-03-11T21:05:46.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:46 smithi157 ceph-mon[33359]: osdmap e644: 8 total, 8 up, 8 in 2024-03-11T21:05:46.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:46 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-128", "mode": "writeback"}]: dispatch 2024-03-11T21:05:47.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:47 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:47.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-128", "mode": "writeback"}]': finished 2024-03-11T21:05:47.726 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:47 smithi129 ceph-mon[30184]: osdmap e645: 8 total, 8 up, 8 in 2024-03-11T21:05:47.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:47 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:47.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-128", "mode": "writeback"}]': finished 2024-03-11T21:05:47.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:47 smithi129 ceph-mon[42066]: osdmap e645: 8 total, 8 up, 8 in 2024-03-11T21:05:47.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:47 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:05:47.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-128", "mode": "writeback"}]': finished 2024-03-11T21:05:47.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:47 smithi157 ceph-mon[33359]: osdmap e645: 8 total, 8 up, 8 in 2024-03-11T21:05:48.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:48 smithi129 ceph-mon[30184]: pgmap v988: 268 pgs: 268 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:05:48.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:48 smithi129 ceph-mon[42066]: pgmap v988: 268 pgs: 268 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:05:48.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:48 smithi157 ceph-mon[33359]: pgmap v988: 268 pgs: 268 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:05:50.225 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:49 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:05:50.226 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:49 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:50.226 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:49 smithi129 ceph-mon[30184]: pgmap v990: 268 pgs: 268 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:05:50.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:49 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:05:50.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:49 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:50.226 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:49 smithi129 ceph-mon[42066]: pgmap v990: 268 pgs: 268 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:05:50.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:49 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:05:50.233 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:49 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:05:50.233 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:49 smithi157 ceph-mon[33359]: pgmap v990: 268 pgs: 268 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:05:52.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:52 smithi129 ceph-mon[30184]: pgmap v991: 268 pgs: 268 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:05:52.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:52 smithi129 ceph-mon[42066]: pgmap v991: 268 pgs: 268 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:05:52.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:52 smithi157 ceph-mon[33359]: pgmap v991: 268 pgs: 268 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:05:53.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:53 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:53.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:53 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:53.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:53 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:05:54.233 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:05:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:05:54.233 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:54 smithi157 ceph-mon[33359]: pgmap v992: 268 pgs: 268 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 259 B/s wr, 1 op/s 2024-03-11T21:05:54.233 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:54.233 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:54 smithi157 ceph-mon[33359]: osdmap e646: 8 total, 8 up, 8 in 2024-03-11T21:05:54.233 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-128"}]: dispatch 2024-03-11T21:05:54.233 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:54 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:05:54.233 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:54 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:05:54.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:54 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:05:54.234 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:54 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:05:54.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:54 smithi129 ceph-mon[30184]: pgmap v992: 268 pgs: 268 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 259 B/s wr, 1 op/s 2024-03-11T21:05:54.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:54.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:54 smithi129 ceph-mon[30184]: osdmap e646: 8 total, 8 up, 8 in 2024-03-11T21:05:54.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-128"}]: dispatch 2024-03-11T21:05:54.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:54 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:05:54.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:54 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:05:54.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:54 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:05:54.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:54 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:05:54.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:54 smithi129 ceph-mon[42066]: pgmap v992: 268 pgs: 268 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 259 B/s wr, 1 op/s 2024-03-11T21:05:54.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:05:54.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:54 smithi129 ceph-mon[42066]: osdmap e646: 8 total, 8 up, 8 in 2024-03-11T21:05:54.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-128"}]: dispatch 2024-03-11T21:05:54.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:54 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:05:54.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:54 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:05:54.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:54 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:05:54.477 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:54 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:05:55.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:55.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:55 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:55.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-128"}]': finished 2024-03-11T21:05:55.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:55 smithi129 ceph-mon[30184]: osdmap e647: 8 total, 8 up, 8 in 2024-03-11T21:05:55.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:55.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:55 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:55.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-128"}]': finished 2024-03-11T21:05:55.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:55 smithi129 ceph-mon[42066]: osdmap e647: 8 total, 8 up, 8 in 2024-03-11T21:05:55.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:05:55.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:55 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:05:55.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-128"}]': finished 2024-03-11T21:05:55.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:55 smithi157 ceph-mon[33359]: osdmap e647: 8 total, 8 up, 8 in 2024-03-11T21:05:56.376 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:05:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:05:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:05:56.377 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:56 smithi129 ceph-mon[30184]: pgmap v995: 268 pgs: 268 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 777 B/s rd, 259 B/s wr, 1 op/s 2024-03-11T21:05:56.377 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:56 smithi129 ceph-mon[30184]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:05:56.377 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:56 smithi129 ceph-mon[30184]: osdmap e648: 8 total, 8 up, 8 in 2024-03-11T21:05:56.377 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:56 smithi129 ceph-mon[42066]: pgmap v995: 268 pgs: 268 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 777 B/s rd, 259 B/s wr, 1 op/s 2024-03-11T21:05:56.377 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:56 smithi129 ceph-mon[42066]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:05:56.377 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:56 smithi129 ceph-mon[42066]: osdmap e648: 8 total, 8 up, 8 in 2024-03-11T21:05:56.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:56 smithi157 ceph-mon[33359]: pgmap v995: 268 pgs: 268 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 777 B/s rd, 259 B/s wr, 1 op/s 2024-03-11T21:05:56.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:56 smithi157 ceph-mon[33359]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:05:56.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:56 smithi157 ceph-mon[33359]: osdmap e648: 8 total, 8 up, 8 in 2024-03-11T21:05:57.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:57 smithi129 ceph-mon[30184]: osdmap e649: 8 total, 8 up, 8 in 2024-03-11T21:05:57.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:57.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:57 smithi129 ceph-mon[42066]: osdmap e649: 8 total, 8 up, 8 in 2024-03-11T21:05:57.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:57.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:57 smithi157 ceph-mon[33359]: osdmap e649: 8 total, 8 up, 8 in 2024-03-11T21:05:57.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:05:58.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:58 smithi129 ceph-mon[30184]: pgmap v998: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:05:58.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:58.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:58 smithi129 ceph-mon[30184]: osdmap e650: 8 total, 8 up, 8 in 2024-03-11T21:05:58.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:58 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:58.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:58 smithi129 ceph-mon[42066]: pgmap v998: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:05:58.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:58.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:58 smithi129 ceph-mon[42066]: osdmap e650: 8 total, 8 up, 8 in 2024-03-11T21:05:58.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:58 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:58.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:58 smithi157 ceph-mon[33359]: pgmap v998: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:05:58.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:05:58.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:58 smithi157 ceph-mon[33359]: osdmap e650: 8 total, 8 up, 8 in 2024-03-11T21:05:58.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:58 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:05:59.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:59.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:59 smithi129 ceph-mon[42066]: osdmap e651: 8 total, 8 up, 8 in 2024-03-11T21:05:59.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-130"}]: dispatch 2024-03-11T21:05:59.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-130"}]': finished 2024-03-11T21:05:59.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:59 smithi129 ceph-mon[42066]: osdmap e652: 8 total, 8 up, 8 in 2024-03-11T21:05:59.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:05:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-130", "mode": "writeback"}]: dispatch 2024-03-11T21:05:59.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:59.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:59 smithi129 ceph-mon[30184]: osdmap e651: 8 total, 8 up, 8 in 2024-03-11T21:05:59.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-130"}]: dispatch 2024-03-11T21:05:59.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-130"}]': finished 2024-03-11T21:05:59.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:59 smithi129 ceph-mon[30184]: osdmap e652: 8 total, 8 up, 8 in 2024-03-11T21:05:59.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:05:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-130", "mode": "writeback"}]: dispatch 2024-03-11T21:05:59.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:05:59.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:59 smithi157 ceph-mon[33359]: osdmap e651: 8 total, 8 up, 8 in 2024-03-11T21:05:59.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-130"}]: dispatch 2024-03-11T21:05:59.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-130"}]': finished 2024-03-11T21:05:59.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:59 smithi157 ceph-mon[33359]: osdmap e652: 8 total, 8 up, 8 in 2024-03-11T21:05:59.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:05:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-130", "mode": "writeback"}]: dispatch 2024-03-11T21:06:00.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:00 smithi129 ceph-mon[42066]: pgmap v1001: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:06:00.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:00 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:06:00.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:00 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-130", "mode": "writeback"}]': finished 2024-03-11T21:06:00.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:00 smithi129 ceph-mon[42066]: osdmap e653: 8 total, 8 up, 8 in 2024-03-11T21:06:00.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:00 smithi129 ceph-mon[30184]: pgmap v1001: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:06:00.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:00 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:06:00.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:00 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-130", "mode": "writeback"}]': finished 2024-03-11T21:06:00.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:00 smithi129 ceph-mon[30184]: osdmap e653: 8 total, 8 up, 8 in 2024-03-11T21:06:00.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:00 smithi157 ceph-mon[33359]: pgmap v1001: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:06:00.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:00 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:06:00.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:00 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-130", "mode": "writeback"}]': finished 2024-03-11T21:06:00.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:00 smithi157 ceph-mon[33359]: osdmap e653: 8 total, 8 up, 8 in 2024-03-11T21:06:01.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:01.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:01.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:02.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:02 smithi129 ceph-mon[30184]: pgmap v1004: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:06:02.827 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:02 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:06:02.827 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:02 smithi129 ceph-mon[30184]: osdmap e654: 8 total, 8 up, 8 in 2024-03-11T21:06:02.827 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:02 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-130"}]: dispatch 2024-03-11T21:06:02.828 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:02 smithi129 ceph-mon[42066]: pgmap v1004: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:06:02.828 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:02 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:06:02.828 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:02 smithi129 ceph-mon[42066]: osdmap e654: 8 total, 8 up, 8 in 2024-03-11T21:06:02.828 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:02 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-130"}]: dispatch 2024-03-11T21:06:02.828 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:02 smithi157 ceph-mon[33359]: pgmap v1004: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 684 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:06:02.828 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:02 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:06:02.828 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:02 smithi157 ceph-mon[33359]: osdmap e654: 8 total, 8 up, 8 in 2024-03-11T21:06:02.829 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:02 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-130"}]: dispatch 2024-03-11T21:06:03.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:03 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:06:03.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:03 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-130"}]': finished 2024-03-11T21:06:03.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:03 smithi129 ceph-mon[30184]: osdmap e655: 8 total, 8 up, 8 in 2024-03-11T21:06:03.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:03 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:06:03.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:03 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-130"}]': finished 2024-03-11T21:06:03.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:03 smithi129 ceph-mon[42066]: osdmap e655: 8 total, 8 up, 8 in 2024-03-11T21:06:03.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:03 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:06:03.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:03 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-130"}]': finished 2024-03-11T21:06:03.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:03 smithi157 ceph-mon[33359]: osdmap e655: 8 total, 8 up, 8 in 2024-03-11T21:06:04.233 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:06:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:06:04.233 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:04 smithi157 ceph-mon[33359]: pgmap v1007: 268 pgs: 268 active+clean; 582 KiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-11T21:06:04.233 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:04 smithi157 ceph-mon[33359]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:06:04.233 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:04 smithi157 ceph-mon[33359]: osdmap e656: 8 total, 8 up, 8 in 2024-03-11T21:06:04.233 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:06:04.233 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:04.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:04 smithi129 ceph-mon[42066]: pgmap v1007: 268 pgs: 268 active+clean; 582 KiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-11T21:06:04.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:04 smithi129 ceph-mon[42066]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:06:04.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:04 smithi129 ceph-mon[42066]: osdmap e656: 8 total, 8 up, 8 in 2024-03-11T21:06:04.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:06:04.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:04.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:04 smithi129 ceph-mon[30184]: pgmap v1007: 268 pgs: 268 active+clean; 582 KiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-11T21:06:04.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:04 smithi129 ceph-mon[30184]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:06:04.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:04 smithi129 ceph-mon[30184]: osdmap e656: 8 total, 8 up, 8 in 2024-03-11T21:06:04.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:06:04.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:05.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:05.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:05 smithi129 ceph-mon[30184]: osdmap e657: 8 total, 8 up, 8 in 2024-03-11T21:06:05.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:06:05.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:05.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:05 smithi129 ceph-mon[42066]: osdmap e657: 8 total, 8 up, 8 in 2024-03-11T21:06:05.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:06:05.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:05.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:05 smithi157 ceph-mon[33359]: osdmap e657: 8 total, 8 up, 8 in 2024-03-11T21:06:05.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:06:06.475 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:06:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:06:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:06:06.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:06 smithi129 ceph-mon[30184]: pgmap v1010: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:06:06.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:06:06.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:06 smithi129 ceph-mon[30184]: osdmap e658: 8 total, 8 up, 8 in 2024-03-11T21:06:06.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:06:06.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:06 smithi129 ceph-mon[42066]: pgmap v1010: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:06:06.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:06:06.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:06 smithi129 ceph-mon[42066]: osdmap e658: 8 total, 8 up, 8 in 2024-03-11T21:06:06.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:06:06.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:06 smithi157 ceph-mon[33359]: pgmap v1010: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-11T21:06:06.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:06:06.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:06 smithi157 ceph-mon[33359]: osdmap e658: 8 total, 8 up, 8 in 2024-03-11T21:06:06.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:06:07.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:06:07.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:07 smithi129 ceph-mon[42066]: osdmap e659: 8 total, 8 up, 8 in 2024-03-11T21:06:07.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-132"}]: dispatch 2024-03-11T21:06:07.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-132"}]': finished 2024-03-11T21:06:07.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:07 smithi129 ceph-mon[42066]: osdmap e660: 8 total, 8 up, 8 in 2024-03-11T21:06:07.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-132", "mode": "writeback"}]: dispatch 2024-03-11T21:06:07.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:06:07.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:07 smithi129 ceph-mon[30184]: osdmap e659: 8 total, 8 up, 8 in 2024-03-11T21:06:07.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-132"}]: dispatch 2024-03-11T21:06:07.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-132"}]': finished 2024-03-11T21:06:07.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:07 smithi129 ceph-mon[30184]: osdmap e660: 8 total, 8 up, 8 in 2024-03-11T21:06:07.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-132", "mode": "writeback"}]: dispatch 2024-03-11T21:06:07.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:06:07.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:07 smithi157 ceph-mon[33359]: osdmap e659: 8 total, 8 up, 8 in 2024-03-11T21:06:07.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-132"}]: dispatch 2024-03-11T21:06:07.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-132"}]': finished 2024-03-11T21:06:07.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:07 smithi157 ceph-mon[33359]: osdmap e660: 8 total, 8 up, 8 in 2024-03-11T21:06:07.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-132", "mode": "writeback"}]: dispatch 2024-03-11T21:06:08.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:08 smithi129 ceph-mon[30184]: pgmap v1013: 268 pgs: 268 active+clean; 582 KiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:06:08.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:08 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:06:08.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-132", "mode": "writeback"}]': finished 2024-03-11T21:06:08.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:08 smithi129 ceph-mon[30184]: osdmap e661: 8 total, 8 up, 8 in 2024-03-11T21:06:08.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:08 smithi129 ceph-mon[42066]: pgmap v1013: 268 pgs: 268 active+clean; 582 KiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:06:08.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:08 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:06:08.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-132", "mode": "writeback"}]': finished 2024-03-11T21:06:08.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:08 smithi129 ceph-mon[42066]: osdmap e661: 8 total, 8 up, 8 in 2024-03-11T21:06:08.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:08 smithi157 ceph-mon[33359]: pgmap v1013: 268 pgs: 268 active+clean; 582 KiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:06:08.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:08 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:06:08.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-132", "mode": "writeback"}]': finished 2024-03-11T21:06:08.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:08 smithi157 ceph-mon[33359]: osdmap e661: 8 total, 8 up, 8 in 2024-03-11T21:06:10.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:10 smithi129 ceph-mon[30184]: pgmap v1016: 268 pgs: 268 active+clean; 582 KiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:10.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:10 smithi129 ceph-mon[30184]: osdmap e662: 8 total, 8 up, 8 in 2024-03-11T21:06:10.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:10 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:06:10.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:10 smithi129 ceph-mon[42066]: pgmap v1016: 268 pgs: 268 active+clean; 582 KiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:10.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:10 smithi129 ceph-mon[42066]: osdmap e662: 8 total, 8 up, 8 in 2024-03-11T21:06:10.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:10 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:06:10.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:10 smithi157 ceph-mon[33359]: pgmap v1016: 268 pgs: 268 active+clean; 582 KiB data, 690 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:10.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:10 smithi157 ceph-mon[33359]: osdmap e662: 8 total, 8 up, 8 in 2024-03-11T21:06:10.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:10 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:06:11.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:11 smithi129 ceph-mon[30184]: osdmap e663: 8 total, 8 up, 8 in 2024-03-11T21:06:11.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:11.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:11 smithi129 ceph-mon[42066]: osdmap e663: 8 total, 8 up, 8 in 2024-03-11T21:06:11.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:11.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:11 smithi157 ceph-mon[33359]: osdmap e663: 8 total, 8 up, 8 in 2024-03-11T21:06:11.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:12.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:12 smithi129 ceph-mon[30184]: pgmap v1019: 268 pgs: 268 active+clean; 582 KiB data, 690 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:06:12.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:06:12.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:12 smithi129 ceph-mon[30184]: osdmap e664: 8 total, 8 up, 8 in 2024-03-11T21:06:12.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-132"}]: dispatch 2024-03-11T21:06:12.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:12 smithi129 ceph-mon[42066]: pgmap v1019: 268 pgs: 268 active+clean; 582 KiB data, 690 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:06:12.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:06:12.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:12 smithi129 ceph-mon[42066]: osdmap e664: 8 total, 8 up, 8 in 2024-03-11T21:06:12.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-132"}]: dispatch 2024-03-11T21:06:12.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:12 smithi157 ceph-mon[33359]: pgmap v1019: 268 pgs: 268 active+clean; 582 KiB data, 690 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:06:12.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:06:12.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:12 smithi157 ceph-mon[33359]: osdmap e664: 8 total, 8 up, 8 in 2024-03-11T21:06:12.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-132"}]: dispatch 2024-03-11T21:06:13.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-132"}]': finished 2024-03-11T21:06:13.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:13 smithi129 ceph-mon[30184]: osdmap e665: 8 total, 8 up, 8 in 2024-03-11T21:06:13.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-132"}]': finished 2024-03-11T21:06:13.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:13 smithi129 ceph-mon[42066]: osdmap e665: 8 total, 8 up, 8 in 2024-03-11T21:06:13.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-132"}]': finished 2024-03-11T21:06:13.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:13 smithi157 ceph-mon[33359]: osdmap e665: 8 total, 8 up, 8 in 2024-03-11T21:06:14.198 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:06:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:06:14.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:14 smithi129 ceph-mon[30184]: pgmap v1022: 268 pgs: 268 active+clean; 582 KiB data, 690 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-11T21:06:14.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:14 smithi129 ceph-mon[30184]: osdmap e666: 8 total, 8 up, 8 in 2024-03-11T21:06:14.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:14.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:14 smithi129 ceph-mon[42066]: pgmap v1022: 268 pgs: 268 active+clean; 582 KiB data, 690 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-11T21:06:14.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:14 smithi129 ceph-mon[42066]: osdmap e666: 8 total, 8 up, 8 in 2024-03-11T21:06:14.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:14.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:14 smithi157 ceph-mon[33359]: pgmap v1022: 268 pgs: 268 active+clean; 582 KiB data, 690 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-11T21:06:14.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:14 smithi157 ceph-mon[33359]: osdmap e666: 8 total, 8 up, 8 in 2024-03-11T21:06:14.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:15.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:15.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:06:15.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:15 smithi129 ceph-mon[30184]: osdmap e667: 8 total, 8 up, 8 in 2024-03-11T21:06:15.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-132"}]: dispatch 2024-03-11T21:06:15.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:15.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:06:15.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:15 smithi129 ceph-mon[42066]: osdmap e667: 8 total, 8 up, 8 in 2024-03-11T21:06:15.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-132"}]: dispatch 2024-03-11T21:06:15.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:15.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:06:15.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:15 smithi157 ceph-mon[33359]: osdmap e667: 8 total, 8 up, 8 in 2024-03-11T21:06:15.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-132"}]: dispatch 2024-03-11T21:06:16.475 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:06:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:06:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:06:16.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:16 smithi129 ceph-mon[42066]: pgmap v1025: 268 pgs: 268 active+clean; 582 KiB data, 690 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-11T21:06:16.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:16 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:06:16.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-132"}]': finished 2024-03-11T21:06:16.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:16 smithi129 ceph-mon[42066]: osdmap e668: 8 total, 8 up, 8 in 2024-03-11T21:06:16.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:16 smithi129 ceph-mon[30184]: pgmap v1025: 268 pgs: 268 active+clean; 582 KiB data, 690 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-11T21:06:16.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:16 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:06:16.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-132"}]': finished 2024-03-11T21:06:16.476 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:16 smithi129 ceph-mon[30184]: osdmap e668: 8 total, 8 up, 8 in 2024-03-11T21:06:16.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:16 smithi157 ceph-mon[33359]: pgmap v1025: 268 pgs: 268 active+clean; 582 KiB data, 690 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-11T21:06:16.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:16 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:06:16.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-132"}]': finished 2024-03-11T21:06:16.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:16 smithi157 ceph-mon[33359]: osdmap e668: 8 total, 8 up, 8 in 2024-03-11T21:06:17.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:17 smithi129 ceph-mon[30184]: osdmap e669: 8 total, 8 up, 8 in 2024-03-11T21:06:17.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:17 smithi129 ceph-mon[42066]: osdmap e669: 8 total, 8 up, 8 in 2024-03-11T21:06:17.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:17 smithi157 ceph-mon[33359]: osdmap e669: 8 total, 8 up, 8 in 2024-03-11T21:06:18.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:18 smithi129 ceph-mon[30184]: pgmap v1028: 236 pgs: 236 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:06:18.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:18 smithi129 ceph-mon[30184]: osdmap e670: 8 total, 8 up, 8 in 2024-03-11T21:06:18.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:06:18.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:18 smithi129 ceph-mon[42066]: pgmap v1028: 236 pgs: 236 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:06:18.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:18 smithi129 ceph-mon[42066]: osdmap e670: 8 total, 8 up, 8 in 2024-03-11T21:06:18.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:06:18.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:18 smithi157 ceph-mon[33359]: pgmap v1028: 236 pgs: 236 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:06:18.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:18 smithi157 ceph-mon[33359]: osdmap e670: 8 total, 8 up, 8 in 2024-03-11T21:06:18.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:06:19.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:06:19.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:19 smithi129 ceph-mon[30184]: osdmap e671: 8 total, 8 up, 8 in 2024-03-11T21:06:19.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:06:19.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:19.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:06:19.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:19 smithi129 ceph-mon[30184]: osdmap e672: 8 total, 8 up, 8 in 2024-03-11T21:06:19.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-134"}]: dispatch 2024-03-11T21:06:19.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:06:19.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:19 smithi129 ceph-mon[42066]: osdmap e671: 8 total, 8 up, 8 in 2024-03-11T21:06:19.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:06:19.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:19.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:06:19.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:19 smithi129 ceph-mon[42066]: osdmap e672: 8 total, 8 up, 8 in 2024-03-11T21:06:19.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-134"}]: dispatch 2024-03-11T21:06:19.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:06:19.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:19 smithi157 ceph-mon[33359]: osdmap e671: 8 total, 8 up, 8 in 2024-03-11T21:06:19.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:06:19.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:19.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:06:19.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:19 smithi157 ceph-mon[33359]: osdmap e672: 8 total, 8 up, 8 in 2024-03-11T21:06:19.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-134"}]: dispatch 2024-03-11T21:06:20.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:20 smithi129 ceph-mon[30184]: pgmap v1031: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:06:20.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-134"}]': finished 2024-03-11T21:06:20.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:20 smithi129 ceph-mon[30184]: osdmap e673: 8 total, 8 up, 8 in 2024-03-11T21:06:20.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-134", "mode": "writeback"}]: dispatch 2024-03-11T21:06:20.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:20 smithi129 ceph-mon[42066]: pgmap v1031: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:06:20.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-134"}]': finished 2024-03-11T21:06:20.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:20 smithi129 ceph-mon[42066]: osdmap e673: 8 total, 8 up, 8 in 2024-03-11T21:06:20.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-134", "mode": "writeback"}]: dispatch 2024-03-11T21:06:20.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:20 smithi157 ceph-mon[33359]: pgmap v1031: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:06:20.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-134"}]': finished 2024-03-11T21:06:20.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:20 smithi157 ceph-mon[33359]: osdmap e673: 8 total, 8 up, 8 in 2024-03-11T21:06:20.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-134", "mode": "writeback"}]: dispatch 2024-03-11T21:06:21.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:21 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:06:21.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-134", "mode": "writeback"}]': finished 2024-03-11T21:06:21.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:21 smithi129 ceph-mon[30184]: osdmap e674: 8 total, 8 up, 8 in 2024-03-11T21:06:21.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:21.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:21 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:06:21.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-134", "mode": "writeback"}]': finished 2024-03-11T21:06:21.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:21 smithi129 ceph-mon[42066]: osdmap e674: 8 total, 8 up, 8 in 2024-03-11T21:06:21.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:21.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:21 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:06:21.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-134", "mode": "writeback"}]': finished 2024-03-11T21:06:21.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:21 smithi157 ceph-mon[33359]: osdmap e674: 8 total, 8 up, 8 in 2024-03-11T21:06:21.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:22.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:22 smithi129 ceph-mon[30184]: pgmap v1034: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:06:22.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:06:22.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:22 smithi129 ceph-mon[30184]: osdmap e675: 8 total, 8 up, 8 in 2024-03-11T21:06:22.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:22 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-134"}]: dispatch 2024-03-11T21:06:22.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:22 smithi129 ceph-mon[42066]: pgmap v1034: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:06:22.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:06:22.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:22 smithi129 ceph-mon[42066]: osdmap e675: 8 total, 8 up, 8 in 2024-03-11T21:06:22.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:22 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-134"}]: dispatch 2024-03-11T21:06:22.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:22 smithi157 ceph-mon[33359]: pgmap v1034: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:06:22.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:06:22.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:22 smithi157 ceph-mon[33359]: osdmap e675: 8 total, 8 up, 8 in 2024-03-11T21:06:22.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:22 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-134"}]: dispatch 2024-03-11T21:06:23.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:23 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:06:23.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:23 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-134"}]': finished 2024-03-11T21:06:23.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:23 smithi129 ceph-mon[30184]: osdmap e676: 8 total, 8 up, 8 in 2024-03-11T21:06:23.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:23 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:06:23.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:23 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-134"}]': finished 2024-03-11T21:06:23.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:23 smithi129 ceph-mon[42066]: osdmap e676: 8 total, 8 up, 8 in 2024-03-11T21:06:23.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:23 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:06:23.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:23 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-134"}]': finished 2024-03-11T21:06:23.734 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:23 smithi157 ceph-mon[33359]: osdmap e676: 8 total, 8 up, 8 in 2024-03-11T21:06:24.232 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:06:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:06:24.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:24 smithi129 ceph-mon[30184]: pgmap v1037: 268 pgs: 268 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-11T21:06:24.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:24 smithi129 ceph-mon[30184]: osdmap e677: 8 total, 8 up, 8 in 2024-03-11T21:06:24.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:24 smithi129 ceph-mon[42066]: pgmap v1037: 268 pgs: 268 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-11T21:06:24.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:24 smithi129 ceph-mon[42066]: osdmap e677: 8 total, 8 up, 8 in 2024-03-11T21:06:24.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:24 smithi157 ceph-mon[33359]: pgmap v1037: 268 pgs: 268 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-11T21:06:24.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:24 smithi157 ceph-mon[33359]: osdmap e677: 8 total, 8 up, 8 in 2024-03-11T21:06:25.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:25.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:25 smithi129 ceph-mon[30184]: osdmap e678: 8 total, 8 up, 8 in 2024-03-11T21:06:25.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:25 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:06:25.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:25.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:25 smithi129 ceph-mon[42066]: osdmap e678: 8 total, 8 up, 8 in 2024-03-11T21:06:25.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:25 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:06:25.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:25.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:25 smithi157 ceph-mon[33359]: osdmap e678: 8 total, 8 up, 8 in 2024-03-11T21:06:25.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:25 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:06:26.475 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:06:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:06:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:06:26.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:26 smithi129 ceph-mon[30184]: pgmap v1040: 236 pgs: 236 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:06:26.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:26 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:06:26.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:26 smithi129 ceph-mon[30184]: osdmap e679: 8 total, 8 up, 8 in 2024-03-11T21:06:26.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:26 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:06:26.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:26 smithi129 ceph-mon[42066]: pgmap v1040: 236 pgs: 236 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:06:26.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:26 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:06:26.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:26 smithi129 ceph-mon[42066]: osdmap e679: 8 total, 8 up, 8 in 2024-03-11T21:06:26.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:26 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:06:26.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:26 smithi157 ceph-mon[33359]: pgmap v1040: 236 pgs: 236 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:06:26.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:26 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:06:26.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:26 smithi157 ceph-mon[33359]: osdmap e679: 8 total, 8 up, 8 in 2024-03-11T21:06:26.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:26 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:06:28.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:28 smithi129 ceph-mon[30184]: pgmap v1043: 268 pgs: 268 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:28.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:06:28.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:28 smithi129 ceph-mon[30184]: osdmap e680: 8 total, 8 up, 8 in 2024-03-11T21:06:28.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-136"}]: dispatch 2024-03-11T21:06:28.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:28 smithi129 ceph-mon[42066]: pgmap v1043: 268 pgs: 268 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:28.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:06:28.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:28 smithi129 ceph-mon[42066]: osdmap e680: 8 total, 8 up, 8 in 2024-03-11T21:06:28.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-136"}]: dispatch 2024-03-11T21:06:28.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:28 smithi157 ceph-mon[33359]: pgmap v1043: 268 pgs: 268 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:28.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:06:28.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:28 smithi157 ceph-mon[33359]: osdmap e680: 8 total, 8 up, 8 in 2024-03-11T21:06:28.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-136"}]: dispatch 2024-03-11T21:06:29.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-136"}]': finished 2024-03-11T21:06:29.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:29 smithi129 ceph-mon[30184]: osdmap e681: 8 total, 8 up, 8 in 2024-03-11T21:06:29.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-136", "mode": "writeback"}]: dispatch 2024-03-11T21:06:29.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:29 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:06:29.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:29 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-136", "mode": "writeback"}]': finished 2024-03-11T21:06:29.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:29 smithi129 ceph-mon[30184]: osdmap e682: 8 total, 8 up, 8 in 2024-03-11T21:06:29.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-136"}]': finished 2024-03-11T21:06:29.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:29 smithi129 ceph-mon[42066]: osdmap e681: 8 total, 8 up, 8 in 2024-03-11T21:06:29.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-136", "mode": "writeback"}]: dispatch 2024-03-11T21:06:29.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:29 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:06:29.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:29 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-136", "mode": "writeback"}]': finished 2024-03-11T21:06:29.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:29 smithi129 ceph-mon[42066]: osdmap e682: 8 total, 8 up, 8 in 2024-03-11T21:06:29.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-136"}]': finished 2024-03-11T21:06:29.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:29 smithi157 ceph-mon[33359]: osdmap e681: 8 total, 8 up, 8 in 2024-03-11T21:06:29.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-136", "mode": "writeback"}]: dispatch 2024-03-11T21:06:29.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:29 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:06:29.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:29 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-136", "mode": "writeback"}]': finished 2024-03-11T21:06:29.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:29 smithi157 ceph-mon[33359]: osdmap e682: 8 total, 8 up, 8 in 2024-03-11T21:06:30.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:30 smithi129 ceph-mon[30184]: pgmap v1046: 268 pgs: 268 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:30.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:30 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:30.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:30 smithi129 ceph-mon[42066]: pgmap v1046: 268 pgs: 268 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:30.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:30 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:30.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:30 smithi157 ceph-mon[33359]: pgmap v1046: 268 pgs: 268 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:30.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:30 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:31.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:06:31.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:31 smithi129 ceph-mon[42066]: osdmap e683: 8 total, 8 up, 8 in 2024-03-11T21:06:31.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:31 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-136"}]: dispatch 2024-03-11T21:06:31.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:06:31.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:31 smithi129 ceph-mon[30184]: osdmap e683: 8 total, 8 up, 8 in 2024-03-11T21:06:31.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:31 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-136"}]: dispatch 2024-03-11T21:06:31.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:06:31.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:31 smithi157 ceph-mon[33359]: osdmap e683: 8 total, 8 up, 8 in 2024-03-11T21:06:31.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:31 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-136"}]: dispatch 2024-03-11T21:06:32.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:32 smithi129 ceph-mon[30184]: pgmap v1049: 268 pgs: 268 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:06:32.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:32 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:06:32.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:32 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-136"}]': finished 2024-03-11T21:06:32.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:32 smithi129 ceph-mon[30184]: osdmap e684: 8 total, 8 up, 8 in 2024-03-11T21:06:32.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:32 smithi129 ceph-mon[42066]: pgmap v1049: 268 pgs: 268 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:06:32.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:32 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:06:32.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:32 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-136"}]': finished 2024-03-11T21:06:32.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:32 smithi129 ceph-mon[42066]: osdmap e684: 8 total, 8 up, 8 in 2024-03-11T21:06:32.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:32 smithi157 ceph-mon[33359]: pgmap v1049: 268 pgs: 268 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:06:32.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:32 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:06:32.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:32 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-136"}]': finished 2024-03-11T21:06:32.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:32 smithi157 ceph-mon[33359]: osdmap e684: 8 total, 8 up, 8 in 2024-03-11T21:06:33.202 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ]c 2024-03-11T21:06:33.203 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.203 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.203 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.203 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.203 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.203 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.203 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.203 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.203 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.203 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.203 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.204 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.204 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.204 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.204 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.204 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.204 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.204 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.204 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.204 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.204 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.204 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.204 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.204 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.205 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.206 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.207 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.208 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.209 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.210 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.210 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.210 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.210 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.210 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.210 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.210 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.210 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.210 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:06:33.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:33 smithi129 ceph-mon[30184]: osdmap e685: 8 total, 8 up, 8 in 2024-03-11T21:06:33.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:33 smithi129 ceph-mon[42066]: osdmap e685: 8 total, 8 up, 8 in 2024-03-11T21:06:33.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:33 smithi157 ceph-mon[33359]: osdmap e685: 8 total, 8 up, 8 in 2024-03-11T21:06:34.232 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:06:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:06:34.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:34 smithi157 ceph-mon[33359]: pgmap v1052: 236 pgs: 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-11T21:06:34.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:34 smithi157 ceph-mon[33359]: osdmap e686: 8 total, 8 up, 8 in 2024-03-11T21:06:34.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:34 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:06:34.233 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:34 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:34.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:34 smithi129 ceph-mon[30184]: pgmap v1052: 236 pgs: 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-11T21:06:34.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:34 smithi129 ceph-mon[30184]: osdmap e686: 8 total, 8 up, 8 in 2024-03-11T21:06:34.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:34 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:06:34.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:34 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:34.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:34 smithi129 ceph-mon[42066]: pgmap v1052: 236 pgs: 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-11T21:06:34.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:34 smithi129 ceph-mon[42066]: osdmap e686: 8 total, 8 up, 8 in 2024-03-11T21:06:34.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:34 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:06:34.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:34 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:35.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:35.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:06:35.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:35 smithi129 ceph-mon[30184]: osdmap e687: 8 total, 8 up, 8 in 2024-03-11T21:06:35.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:06:35.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:06:35.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:35 smithi129 ceph-mon[30184]: osdmap e688: 8 total, 8 up, 8 in 2024-03-11T21:06:35.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:06:35.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:35.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:06:35.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:35 smithi129 ceph-mon[42066]: osdmap e687: 8 total, 8 up, 8 in 2024-03-11T21:06:35.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:06:35.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:06:35.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:35 smithi129 ceph-mon[42066]: osdmap e688: 8 total, 8 up, 8 in 2024-03-11T21:06:35.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:06:35.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:35.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:06:35.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:35 smithi157 ceph-mon[33359]: osdmap e687: 8 total, 8 up, 8 in 2024-03-11T21:06:35.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:06:35.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:06:35.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:35 smithi157 ceph-mon[33359]: osdmap e688: 8 total, 8 up, 8 in 2024-03-11T21:06:35.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:06:36.474 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:06:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:06:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:06:36.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:36 smithi129 ceph-mon[30184]: pgmap v1055: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-11T21:06:36.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:06:36.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:36 smithi129 ceph-mon[30184]: osdmap e689: 8 total, 8 up, 8 in 2024-03-11T21:06:36.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:06:36.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:36 smithi129 ceph-mon[42066]: pgmap v1055: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-11T21:06:36.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:06:36.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:36 smithi129 ceph-mon[42066]: osdmap e689: 8 total, 8 up, 8 in 2024-03-11T21:06:36.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:06:36.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:36 smithi157 ceph-mon[33359]: pgmap v1055: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-11T21:06:36.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:06:36.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:36 smithi157 ceph-mon[33359]: osdmap e689: 8 total, 8 up, 8 in 2024-03-11T21:06:36.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:06:38.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:38 smithi129 ceph-mon[30184]: pgmap v1058: 268 pgs: 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:38.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:06:38.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:38 smithi129 ceph-mon[30184]: osdmap e690: 8 total, 8 up, 8 in 2024-03-11T21:06:38.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:06:38.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:38 smithi129 ceph-mon[42066]: pgmap v1058: 268 pgs: 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:38.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:06:38.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:38 smithi129 ceph-mon[42066]: osdmap e690: 8 total, 8 up, 8 in 2024-03-11T21:06:38.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:06:38.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:38 smithi157 ceph-mon[33359]: pgmap v1058: 268 pgs: 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:38.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:06:38.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:38 smithi157 ceph-mon[33359]: osdmap e690: 8 total, 8 up, 8 in 2024-03-11T21:06:38.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-11T21:06:39.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:06:39.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:39 smithi129 ceph-mon[30184]: osdmap e691: 8 total, 8 up, 8 in 2024-03-11T21:06:39.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:06:39.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:39 smithi129 ceph-mon[42066]: osdmap e691: 8 total, 8 up, 8 in 2024-03-11T21:06:39.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-11T21:06:39.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:39 smithi157 ceph-mon[33359]: osdmap e691: 8 total, 8 up, 8 in 2024-03-11T21:06:40.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:40 smithi129 ceph-mon[30184]: pgmap v1061: 268 pgs: 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:40.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:40.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-138"}]: dispatch 2024-03-11T21:06:40.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:40 smithi129 ceph-mon[42066]: pgmap v1061: 268 pgs: 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:40.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:40.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-138"}]: dispatch 2024-03-11T21:06:40.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:40 smithi157 ceph-mon[33359]: pgmap v1061: 268 pgs: 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:40.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:06:40.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-138"}]: dispatch 2024-03-11T21:06:41.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:41 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-138"}]': finished 2024-03-11T21:06:41.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:41 smithi129 ceph-mon[30184]: osdmap e692: 8 total, 8 up, 8 in 2024-03-11T21:06:41.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:41 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-138"}]': finished 2024-03-11T21:06:41.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:41 smithi129 ceph-mon[42066]: osdmap e692: 8 total, 8 up, 8 in 2024-03-11T21:06:41.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:41 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-138"}]': finished 2024-03-11T21:06:41.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:41 smithi157 ceph-mon[33359]: osdmap e692: 8 total, 8 up, 8 in 2024-03-11T21:06:42.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:42 smithi129 ceph-mon[30184]: pgmap v1063: 268 pgs: 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:06:42.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:42 smithi129 ceph-mon[30184]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:06:42.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:42 smithi129 ceph-mon[30184]: osdmap e693: 8 total, 8 up, 8 in 2024-03-11T21:06:42.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:42 smithi129 ceph-mon[42066]: pgmap v1063: 268 pgs: 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:06:42.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:42 smithi129 ceph-mon[42066]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:06:42.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:42 smithi129 ceph-mon[42066]: osdmap e693: 8 total, 8 up, 8 in 2024-03-11T21:06:42.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:42 smithi157 ceph-mon[33359]: pgmap v1063: 268 pgs: 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:06:42.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:42 smithi157 ceph-mon[33359]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:06:42.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:42 smithi157 ceph-mon[33359]: osdmap e693: 8 total, 8 up, 8 in 2024-03-11T21:06:43.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:43 smithi129 ceph-mon[30184]: osdmap e694: 8 total, 8 up, 8 in 2024-03-11T21:06:43.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:06:43.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:43 smithi129 ceph-mon[42066]: osdmap e694: 8 total, 8 up, 8 in 2024-03-11T21:06:43.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:06:43.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:43 smithi157 ceph-mon[33359]: osdmap e694: 8 total, 8 up, 8 in 2024-03-11T21:06:43.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:06:44.232 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:06:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:06:44.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:44 smithi157 ceph-mon[33359]: pgmap v1066: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:06:44.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:06:44.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:44 smithi157 ceph-mon[33359]: osdmap e695: 8 total, 8 up, 8 in 2024-03-11T21:06:44.233 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:06:44.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:44 smithi129 ceph-mon[30184]: pgmap v1066: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:06:44.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:06:44.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:44 smithi129 ceph-mon[30184]: osdmap e695: 8 total, 8 up, 8 in 2024-03-11T21:06:44.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:06:44.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:44 smithi129 ceph-mon[42066]: pgmap v1066: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:06:44.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:06:44.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:44 smithi129 ceph-mon[42066]: osdmap e695: 8 total, 8 up, 8 in 2024-03-11T21:06:44.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:06:45.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:45.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:06:45.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:45 smithi129 ceph-mon[30184]: osdmap e696: 8 total, 8 up, 8 in 2024-03-11T21:06:45.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T21:06:45.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_count","val": "3"}]': finished 2024-03-11T21:06:45.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:45.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:06:45.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:45 smithi129 ceph-mon[42066]: osdmap e696: 8 total, 8 up, 8 in 2024-03-11T21:06:45.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T21:06:45.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_count","val": "3"}]': finished 2024-03-11T21:06:45.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:45.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:06:45.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:45 smithi157 ceph-mon[33359]: osdmap e696: 8 total, 8 up, 8 in 2024-03-11T21:06:45.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-11T21:06:45.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_count","val": "3"}]': finished 2024-03-11T21:06:46.474 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:06:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:06:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:06:46.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:46 smithi129 ceph-mon[30184]: pgmap v1069: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:46.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:46 smithi129 ceph-mon[30184]: osdmap e697: 8 total, 8 up, 8 in 2024-03-11T21:06:46.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T21:06:46.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_period","val": "3"}]': finished 2024-03-11T21:06:46.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:46 smithi129 ceph-mon[30184]: osdmap e698: 8 total, 8 up, 8 in 2024-03-11T21:06:46.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:46 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:06:46.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:46 smithi129 ceph-mon[42066]: pgmap v1069: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:46.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:46 smithi129 ceph-mon[42066]: osdmap e697: 8 total, 8 up, 8 in 2024-03-11T21:06:46.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:46 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T21:06:46.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:46 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_period","val": "3"}]': finished 2024-03-11T21:06:46.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:46 smithi129 ceph-mon[42066]: osdmap e698: 8 total, 8 up, 8 in 2024-03-11T21:06:46.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:46 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:06:46.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:46 smithi157 ceph-mon[33359]: pgmap v1069: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:46.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:46 smithi157 ceph-mon[33359]: osdmap e697: 8 total, 8 up, 8 in 2024-03-11T21:06:46.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:46 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-11T21:06:46.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:46 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_period","val": "3"}]': finished 2024-03-11T21:06:46.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:46 smithi157 ceph-mon[33359]: osdmap e698: 8 total, 8 up, 8 in 2024-03-11T21:06:46.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:46 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:06:48.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:48 smithi129 ceph-mon[30184]: pgmap v1072: 268 pgs: 268 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:48.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:06:48.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:48 smithi129 ceph-mon[30184]: osdmap e699: 8 total, 8 up, 8 in 2024-03-11T21:06:48.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T21:06:48.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:48 smithi129 ceph-mon[42066]: pgmap v1072: 268 pgs: 268 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:48.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:06:48.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:48 smithi129 ceph-mon[42066]: osdmap e699: 8 total, 8 up, 8 in 2024-03-11T21:06:48.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T21:06:48.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:48 smithi157 ceph-mon[33359]: pgmap v1072: 268 pgs: 268 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:48.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:06:48.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:48 smithi157 ceph-mon[33359]: osdmap e699: 8 total, 8 up, 8 in 2024-03-11T21:06:48.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-11T21:06:49.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-11T21:06:49.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:49 smithi129 ceph-mon[30184]: osdmap e700: 8 total, 8 up, 8 in 2024-03-11T21:06:49.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:49 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:49.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-11T21:06:49.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:49 smithi129 ceph-mon[42066]: osdmap e700: 8 total, 8 up, 8 in 2024-03-11T21:06:49.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:49 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:49.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-11T21:06:49.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:49 smithi157 ceph-mon[33359]: osdmap e700: 8 total, 8 up, 8 in 2024-03-11T21:06:49.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:49 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:06:50.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:50 smithi129 ceph-mon[42066]: pgmap v1075: 268 pgs: 268 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:50.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:50 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:06:50.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:50 smithi129 ceph-mon[30184]: pgmap v1075: 268 pgs: 268 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:50.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:50 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:06:50.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:50 smithi157 ceph-mon[33359]: pgmap v1075: 268 pgs: 268 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:06:50.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:50 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:06:52.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:52 smithi129 ceph-mon[30184]: pgmap v1076: 268 pgs: 268 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 876 B/s rd, 0 op/s 2024-03-11T21:06:52.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:52 smithi129 ceph-mon[42066]: pgmap v1076: 268 pgs: 268 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 876 B/s rd, 0 op/s 2024-03-11T21:06:52.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:52 smithi157 ceph-mon[33359]: pgmap v1076: 268 pgs: 268 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 876 B/s rd, 0 op/s 2024-03-11T21:06:54.232 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:06:53 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:06:54.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:54 smithi157 ceph-mon[33359]: pgmap v1077: 268 pgs: 268 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 4.8 KiB/s wr, 2 op/s 2024-03-11T21:06:54.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:54 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:06:54.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:54 smithi129 ceph-mon[30184]: pgmap v1077: 268 pgs: 268 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 4.8 KiB/s wr, 2 op/s 2024-03-11T21:06:54.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:54 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:06:54.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:54 smithi129 ceph-mon[42066]: pgmap v1077: 268 pgs: 268 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 4.8 KiB/s wr, 2 op/s 2024-03-11T21:06:54.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:54 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:06:55.975 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:55.975 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:55 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:06:55.975 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:55 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:06:55.975 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:55 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:06:55.975 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:55 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:06:55.975 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:55 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:06:55.975 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:55.976 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:55 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:06:55.976 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:55 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:06:55.976 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:55 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:06:55.976 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:55 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:06:55.976 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:55 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:06:55.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:06:55.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:55 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:06:55.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:55 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:06:55.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:55 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:06:55.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:55 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:06:55.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:55 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:06:56.474 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:06:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:06:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:06:56.974 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:56 smithi129 ceph-mon[42066]: pgmap v1078: 268 pgs: 268 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 4.1 KiB/s wr, 1 op/s 2024-03-11T21:06:56.974 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:56 smithi129 ceph-mon[30184]: pgmap v1078: 268 pgs: 268 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 4.1 KiB/s wr, 1 op/s 2024-03-11T21:06:56.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:56 smithi157 ceph-mon[33359]: pgmap v1078: 268 pgs: 268 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 4.1 KiB/s wr, 1 op/s 2024-03-11T21:06:58.974 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:06:58 smithi129 ceph-mon[30184]: pgmap v1079: 268 pgs: 268 active+clean; 4.5 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 7.5 KiB/s wr, 2 op/s 2024-03-11T21:06:58.974 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:06:58 smithi129 ceph-mon[42066]: pgmap v1079: 268 pgs: 268 active+clean; 4.5 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 7.5 KiB/s wr, 2 op/s 2024-03-11T21:06:58.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:06:58 smithi157 ceph-mon[33359]: pgmap v1079: 268 pgs: 268 active+clean; 4.5 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 7.5 KiB/s wr, 2 op/s 2024-03-11T21:07:00.974 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:00 smithi129 ceph-mon[30184]: pgmap v1080: 268 pgs: 268 active+clean; 4.5 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 944 B/s rd, 6.8 KiB/s wr, 1 op/s 2024-03-11T21:07:00.974 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:00 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:00.974 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:00 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-140"}]: dispatch 2024-03-11T21:07:00.975 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:00 smithi129 ceph-mon[42066]: pgmap v1080: 268 pgs: 268 active+clean; 4.5 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 944 B/s rd, 6.8 KiB/s wr, 1 op/s 2024-03-11T21:07:00.975 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:00 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:00.975 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:00 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-140"}]: dispatch 2024-03-11T21:07:00.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:00 smithi157 ceph-mon[33359]: pgmap v1080: 268 pgs: 268 active+clean; 4.5 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 944 B/s rd, 6.8 KiB/s wr, 1 op/s 2024-03-11T21:07:00.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:00 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:00.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:00 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-140"}]: dispatch 2024-03-11T21:07:02.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:02 smithi129 ceph-mon[30184]: pgmap v1081: 268 pgs: 268 active+clean; 4.5 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 6.2 KiB/s wr, 1 op/s 2024-03-11T21:07:02.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:02 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-140"}]': finished 2024-03-11T21:07:02.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:02 smithi129 ceph-mon[30184]: osdmap e701: 8 total, 8 up, 8 in 2024-03-11T21:07:02.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:02 smithi129 ceph-mon[42066]: pgmap v1081: 268 pgs: 268 active+clean; 4.5 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 6.2 KiB/s wr, 1 op/s 2024-03-11T21:07:02.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:02 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-140"}]': finished 2024-03-11T21:07:02.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:02 smithi129 ceph-mon[42066]: osdmap e701: 8 total, 8 up, 8 in 2024-03-11T21:07:02.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:02 smithi157 ceph-mon[33359]: pgmap v1081: 268 pgs: 268 active+clean; 4.5 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 6.2 KiB/s wr, 1 op/s 2024-03-11T21:07:02.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:02 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-140"}]': finished 2024-03-11T21:07:02.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:02 smithi157 ceph-mon[33359]: osdmap e701: 8 total, 8 up, 8 in 2024-03-11T21:07:03.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:03 smithi129 ceph-mon[30184]: osdmap e702: 8 total, 8 up, 8 in 2024-03-11T21:07:03.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:03 smithi129 ceph-mon[42066]: osdmap e702: 8 total, 8 up, 8 in 2024-03-11T21:07:03.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:03 smithi157 ceph-mon[33359]: osdmap e702: 8 total, 8 up, 8 in 2024-03-11T21:07:04.232 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:07:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:07:04.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:04 smithi157 ceph-mon[33359]: pgmap v1084: 236 pgs: 236 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:04.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:04 smithi157 ceph-mon[33359]: osdmap e703: 8 total, 8 up, 8 in 2024-03-11T21:07:04.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:04.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:04.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:04 smithi129 ceph-mon[30184]: pgmap v1084: 236 pgs: 236 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:04.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:04 smithi129 ceph-mon[30184]: osdmap e703: 8 total, 8 up, 8 in 2024-03-11T21:07:04.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:04.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:04.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:04 smithi129 ceph-mon[42066]: pgmap v1084: 236 pgs: 236 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:04.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:04 smithi129 ceph-mon[42066]: osdmap e703: 8 total, 8 up, 8 in 2024-03-11T21:07:04.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:04.476 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:05.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:05.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:07:05.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:05 smithi129 ceph-mon[30184]: osdmap e704: 8 total, 8 up, 8 in 2024-03-11T21:07:05.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:07:05.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:05.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:07:05.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:05 smithi129 ceph-mon[42066]: osdmap e704: 8 total, 8 up, 8 in 2024-03-11T21:07:05.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:07:05.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:05.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:07:05.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:05 smithi157 ceph-mon[33359]: osdmap e704: 8 total, 8 up, 8 in 2024-03-11T21:07:05.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:07:06.474 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:07:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:07:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:07:06.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:06 smithi129 ceph-mon[30184]: pgmap v1087: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:06.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:07:06.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:06 smithi129 ceph-mon[30184]: osdmap e705: 8 total, 8 up, 8 in 2024-03-11T21:07:06.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:06 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-142"}]: dispatch 2024-03-11T21:07:06.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:06 smithi129 ceph-mon[42066]: pgmap v1087: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:06.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:07:06.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:06 smithi129 ceph-mon[42066]: osdmap e705: 8 total, 8 up, 8 in 2024-03-11T21:07:06.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:06 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-142"}]: dispatch 2024-03-11T21:07:06.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:06 smithi157 ceph-mon[33359]: pgmap v1087: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:06.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:07:06.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:06 smithi157 ceph-mon[33359]: osdmap e705: 8 total, 8 up, 8 in 2024-03-11T21:07:06.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:06 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-142"}]: dispatch 2024-03-11T21:07:07.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-142"}]': finished 2024-03-11T21:07:07.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:07 smithi129 ceph-mon[30184]: osdmap e706: 8 total, 8 up, 8 in 2024-03-11T21:07:07.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-142", "mode": "writeback"}]: dispatch 2024-03-11T21:07:07.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-142"}]': finished 2024-03-11T21:07:07.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:07 smithi129 ceph-mon[42066]: osdmap e706: 8 total, 8 up, 8 in 2024-03-11T21:07:07.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-142", "mode": "writeback"}]: dispatch 2024-03-11T21:07:07.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-142"}]': finished 2024-03-11T21:07:07.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:07 smithi157 ceph-mon[33359]: osdmap e706: 8 total, 8 up, 8 in 2024-03-11T21:07:07.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-142", "mode": "writeback"}]: dispatch 2024-03-11T21:07:08.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:08 smithi129 ceph-mon[30184]: pgmap v1090: 268 pgs: 268 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-11T21:07:08.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:08 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:07:08.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-142", "mode": "writeback"}]': finished 2024-03-11T21:07:08.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:08 smithi129 ceph-mon[30184]: osdmap e707: 8 total, 8 up, 8 in 2024-03-11T21:07:08.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:07:08.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:08 smithi129 ceph-mon[42066]: pgmap v1090: 268 pgs: 268 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-11T21:07:08.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:08 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:07:08.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-142", "mode": "writeback"}]': finished 2024-03-11T21:07:08.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:08 smithi129 ceph-mon[42066]: osdmap e707: 8 total, 8 up, 8 in 2024-03-11T21:07:08.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:07:08.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:08 smithi157 ceph-mon[33359]: pgmap v1090: 268 pgs: 268 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-11T21:07:08.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:08 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:07:08.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-142", "mode": "writeback"}]': finished 2024-03-11T21:07:08.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:08 smithi157 ceph-mon[33359]: osdmap e707: 8 total, 8 up, 8 in 2024-03-11T21:07:08.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:07:09.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:07:09.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:09 smithi129 ceph-mon[30184]: osdmap e708: 8 total, 8 up, 8 in 2024-03-11T21:07:09.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:07:09.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:07:09.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:09 smithi129 ceph-mon[42066]: osdmap e708: 8 total, 8 up, 8 in 2024-03-11T21:07:09.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:07:09.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:07:09.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:09 smithi157 ceph-mon[33359]: osdmap e708: 8 total, 8 up, 8 in 2024-03-11T21:07:09.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:07:10.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:10 smithi129 ceph-mon[30184]: pgmap v1093: 268 pgs: 268 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-11T21:07:10.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:07:10.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:10 smithi129 ceph-mon[30184]: osdmap e709: 8 total, 8 up, 8 in 2024-03-11T21:07:10.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:07:10.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:10 smithi129 ceph-mon[42066]: pgmap v1093: 268 pgs: 268 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-11T21:07:10.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:07:10.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:10 smithi129 ceph-mon[42066]: osdmap e709: 8 total, 8 up, 8 in 2024-03-11T21:07:10.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:07:10.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:10 smithi157 ceph-mon[33359]: pgmap v1093: 268 pgs: 268 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-11T21:07:10.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:07:10.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:10 smithi157 ceph-mon[33359]: osdmap e709: 8 total, 8 up, 8 in 2024-03-11T21:07:10.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:07:11.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:11 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:07:11.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:07:11.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:11 smithi129 ceph-mon[30184]: osdmap e710: 8 total, 8 up, 8 in 2024-03-11T21:07:11.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:07:11.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:07:11.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:11 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:07:11.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:07:11.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:11 smithi129 ceph-mon[42066]: osdmap e710: 8 total, 8 up, 8 in 2024-03-11T21:07:11.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:07:11.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:07:11.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:11 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:07:11.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:07:11.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:11 smithi157 ceph-mon[33359]: osdmap e710: 8 total, 8 up, 8 in 2024-03-11T21:07:11.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:07:11.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:07:12.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:12 smithi129 ceph-mon[30184]: pgmap v1096: 268 pgs: 268 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:07:12.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:12 smithi129 ceph-mon[30184]: osdmap e711: 8 total, 8 up, 8 in 2024-03-11T21:07:12.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T21:07:12.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-11T21:07:12.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:12 smithi129 ceph-mon[30184]: osdmap e712: 8 total, 8 up, 8 in 2024-03-11T21:07:12.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T21:07:12.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:12 smithi129 ceph-mon[42066]: pgmap v1096: 268 pgs: 268 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:07:12.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:12 smithi129 ceph-mon[42066]: osdmap e711: 8 total, 8 up, 8 in 2024-03-11T21:07:12.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T21:07:12.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-11T21:07:12.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:12 smithi129 ceph-mon[42066]: osdmap e712: 8 total, 8 up, 8 in 2024-03-11T21:07:12.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T21:07:12.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:12 smithi157 ceph-mon[33359]: pgmap v1096: 268 pgs: 268 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:07:12.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:12 smithi157 ceph-mon[33359]: osdmap e711: 8 total, 8 up, 8 in 2024-03-11T21:07:12.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-11T21:07:12.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-11T21:07:12.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:12 smithi157 ceph-mon[33359]: osdmap e712: 8 total, 8 up, 8 in 2024-03-11T21:07:12.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-11T21:07:14.231 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:07:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:07:14.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:14 smithi157 ceph-mon[33359]: pgmap v1099: 268 pgs: 268 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:14.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-11T21:07:14.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:14 smithi157 ceph-mon[33359]: osdmap e713: 8 total, 8 up, 8 in 2024-03-11T21:07:14.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:14.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:14 smithi129 ceph-mon[30184]: pgmap v1099: 268 pgs: 268 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:14.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-11T21:07:14.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:14 smithi129 ceph-mon[30184]: osdmap e713: 8 total, 8 up, 8 in 2024-03-11T21:07:14.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:14.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:14 smithi129 ceph-mon[42066]: pgmap v1099: 268 pgs: 268 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:14.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-11T21:07:14.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:14 smithi129 ceph-mon[42066]: osdmap e713: 8 total, 8 up, 8 in 2024-03-11T21:07:14.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:15.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:15.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:07:15.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:15 smithi129 ceph-mon[30184]: osdmap e714: 8 total, 8 up, 8 in 2024-03-11T21:07:15.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:15 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-142"}]: dispatch 2024-03-11T21:07:15.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:15.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:07:15.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:15 smithi129 ceph-mon[42066]: osdmap e714: 8 total, 8 up, 8 in 2024-03-11T21:07:15.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:15 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-142"}]: dispatch 2024-03-11T21:07:15.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:15.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:07:15.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:15 smithi157 ceph-mon[33359]: osdmap e714: 8 total, 8 up, 8 in 2024-03-11T21:07:15.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:15 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-142"}]: dispatch 2024-03-11T21:07:16.474 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:07:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:07:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:07:16.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:16 smithi129 ceph-mon[30184]: pgmap v1102: 268 pgs: 268 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:16.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-142"}]': finished 2024-03-11T21:07:16.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:16 smithi129 ceph-mon[30184]: osdmap e715: 8 total, 8 up, 8 in 2024-03-11T21:07:16.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:16.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:16 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-142"}]: dispatch 2024-03-11T21:07:16.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:16 smithi129 ceph-mon[42066]: pgmap v1102: 268 pgs: 268 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:16.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-142"}]': finished 2024-03-11T21:07:16.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:16 smithi129 ceph-mon[42066]: osdmap e715: 8 total, 8 up, 8 in 2024-03-11T21:07:16.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:16.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:16 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-142"}]: dispatch 2024-03-11T21:07:16.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:16 smithi157 ceph-mon[33359]: pgmap v1102: 268 pgs: 268 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:07:16.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-142"}]': finished 2024-03-11T21:07:16.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:16 smithi157 ceph-mon[33359]: osdmap e715: 8 total, 8 up, 8 in 2024-03-11T21:07:16.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:16.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:16 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-142"}]: dispatch 2024-03-11T21:07:17.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:17 smithi129 ceph-mon[30184]: osdmap e716: 8 total, 8 up, 8 in 2024-03-11T21:07:17.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:17 smithi129 ceph-mon[42066]: osdmap e716: 8 total, 8 up, 8 in 2024-03-11T21:07:17.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:17 smithi157 ceph-mon[33359]: osdmap e716: 8 total, 8 up, 8 in 2024-03-11T21:07:18.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:18 smithi129 ceph-mon[30184]: pgmap v1105: 236 pgs: 236 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:07:18.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:18 smithi129 ceph-mon[30184]: osdmap e717: 8 total, 8 up, 8 in 2024-03-11T21:07:18.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:18 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:18.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:18 smithi129 ceph-mon[42066]: pgmap v1105: 236 pgs: 236 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:07:18.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:18 smithi129 ceph-mon[42066]: osdmap e717: 8 total, 8 up, 8 in 2024-03-11T21:07:18.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:18 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:18.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:18 smithi157 ceph-mon[33359]: pgmap v1105: 236 pgs: 236 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:07:18.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:18 smithi157 ceph-mon[33359]: osdmap e717: 8 total, 8 up, 8 in 2024-03-11T21:07:18.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:18 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:19.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:07:19.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:19 smithi129 ceph-mon[30184]: osdmap e718: 8 total, 8 up, 8 in 2024-03-11T21:07:19.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:19 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:07:19.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:19.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:07:19.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:19 smithi129 ceph-mon[42066]: osdmap e718: 8 total, 8 up, 8 in 2024-03-11T21:07:19.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:19 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:07:19.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:19.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:07:19.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:19 smithi157 ceph-mon[33359]: osdmap e718: 8 total, 8 up, 8 in 2024-03-11T21:07:19.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:19 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:07:19.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:20.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:20 smithi129 ceph-mon[42066]: pgmap v1108: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:07:20.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:07:20.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:20 smithi129 ceph-mon[42066]: osdmap e719: 8 total, 8 up, 8 in 2024-03-11T21:07:20.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:20 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-144"}]: dispatch 2024-03-11T21:07:20.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:20 smithi129 ceph-mon[30184]: pgmap v1108: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:07:20.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:07:20.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:20 smithi129 ceph-mon[30184]: osdmap e719: 8 total, 8 up, 8 in 2024-03-11T21:07:20.475 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:20 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-144"}]: dispatch 2024-03-11T21:07:20.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:20 smithi157 ceph-mon[33359]: pgmap v1108: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:07:20.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:07:20.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:20 smithi157 ceph-mon[33359]: osdmap e719: 8 total, 8 up, 8 in 2024-03-11T21:07:20.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:20 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-144"}]: dispatch 2024-03-11T21:07:21.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-144"}]': finished 2024-03-11T21:07:21.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:21 smithi129 ceph-mon[30184]: osdmap e720: 8 total, 8 up, 8 in 2024-03-11T21:07:21.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-144", "mode": "readproxy"}]: dispatch 2024-03-11T21:07:21.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:21 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:07:21.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:21 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-144", "mode": "readproxy"}]': finished 2024-03-11T21:07:21.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-144"}]': finished 2024-03-11T21:07:21.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:21 smithi129 ceph-mon[42066]: osdmap e720: 8 total, 8 up, 8 in 2024-03-11T21:07:21.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-144", "mode": "readproxy"}]: dispatch 2024-03-11T21:07:21.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:21 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:07:21.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:21 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-144", "mode": "readproxy"}]': finished 2024-03-11T21:07:21.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-144"}]': finished 2024-03-11T21:07:21.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:21 smithi157 ceph-mon[33359]: osdmap e720: 8 total, 8 up, 8 in 2024-03-11T21:07:21.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-144", "mode": "readproxy"}]: dispatch 2024-03-11T21:07:21.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:21 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:07:21.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:21 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-144", "mode": "readproxy"}]': finished 2024-03-11T21:07:22.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:22 smithi129 ceph-mon[30184]: pgmap v1111: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:07:22.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:22 smithi129 ceph-mon[30184]: osdmap e721: 8 total, 8 up, 8 in 2024-03-11T21:07:22.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:22 smithi129 ceph-mon[42066]: pgmap v1111: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:07:22.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:22 smithi129 ceph-mon[42066]: osdmap e721: 8 total, 8 up, 8 in 2024-03-11T21:07:22.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:22 smithi157 ceph-mon[33359]: pgmap v1111: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:07:22.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:22 smithi157 ceph-mon[33359]: osdmap e721: 8 total, 8 up, 8 in 2024-03-11T21:07:24.231 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:07:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:07:24.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:24 smithi157 ceph-mon[33359]: pgmap v1113: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 210 B/s wr, 1 op/s 2024-03-11T21:07:24.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:24 smithi129 ceph-mon[30184]: pgmap v1113: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 210 B/s wr, 1 op/s 2024-03-11T21:07:24.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:24 smithi129 ceph-mon[42066]: pgmap v1113: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 210 B/s wr, 1 op/s 2024-03-11T21:07:25.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:25.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:25.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:26.375 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:26 smithi129 ceph-mon[30184]: pgmap v1114: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-11T21:07:26.375 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:07:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:07:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:07:26.376 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:26 smithi129 ceph-mon[42066]: pgmap v1114: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-11T21:07:26.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:26 smithi157 ceph-mon[33359]: pgmap v1114: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-11T21:07:28.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:28 smithi129 ceph-mon[30184]: pgmap v1115: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 130 B/s wr, 1 op/s 2024-03-11T21:07:28.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:28 smithi129 ceph-mon[42066]: pgmap v1115: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 130 B/s wr, 1 op/s 2024-03-11T21:07:28.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:28 smithi157 ceph-mon[33359]: pgmap v1115: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 130 B/s wr, 1 op/s 2024-03-11T21:07:30.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:30 smithi129 ceph-mon[30184]: pgmap v1116: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 B/s wr, 1 op/s 2024-03-11T21:07:30.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:30 smithi129 ceph-mon[42066]: pgmap v1116: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 B/s wr, 1 op/s 2024-03-11T21:07:30.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:30 smithi157 ceph-mon[33359]: pgmap v1116: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 B/s wr, 1 op/s 2024-03-11T21:07:32.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:32 smithi129 ceph-mon[30184]: pgmap v1117: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-11T21:07:32.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:32 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:32.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:32 smithi129 ceph-mon[42066]: pgmap v1117: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-11T21:07:32.475 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:32 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:32.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:32 smithi157 ceph-mon[33359]: pgmap v1117: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-11T21:07:32.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:32 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:33.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:07:33.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:33 smithi157 ceph-mon[33359]: osdmap e722: 8 total, 8 up, 8 in 2024-03-11T21:07:33.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-144"}]: dispatch 2024-03-11T21:07:33.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:07:33.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:33 smithi129 ceph-mon[30184]: osdmap e722: 8 total, 8 up, 8 in 2024-03-11T21:07:33.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-144"}]: dispatch 2024-03-11T21:07:33.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:07:33.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:33 smithi129 ceph-mon[42066]: osdmap e722: 8 total, 8 up, 8 in 2024-03-11T21:07:33.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-144"}]: dispatch 2024-03-11T21:07:34.231 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:07:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:07:34.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:34 smithi157 ceph-mon[33359]: pgmap v1119: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:07:34.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:34 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:07:34.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:34 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-144"}]': finished 2024-03-11T21:07:34.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:34 smithi157 ceph-mon[33359]: osdmap e723: 8 total, 8 up, 8 in 2024-03-11T21:07:34.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:34 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:34.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:34 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-144"}]: dispatch 2024-03-11T21:07:34.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:34 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:34.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:34 smithi129 ceph-mon[42066]: pgmap v1119: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:07:34.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:34 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:07:34.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:34 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-144"}]': finished 2024-03-11T21:07:34.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:34 smithi129 ceph-mon[42066]: osdmap e723: 8 total, 8 up, 8 in 2024-03-11T21:07:34.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:34 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:34.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:34 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-144"}]: dispatch 2024-03-11T21:07:34.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:34 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:34.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:34 smithi129 ceph-mon[30184]: pgmap v1119: 268 pgs: 268 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:07:34.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:34 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:07:34.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:34 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-144"}]': finished 2024-03-11T21:07:34.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:34 smithi129 ceph-mon[30184]: osdmap e723: 8 total, 8 up, 8 in 2024-03-11T21:07:34.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:34 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:34.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:34 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-144"}]: dispatch 2024-03-11T21:07:34.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:34 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:35.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:35.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:35 smithi129 ceph-mon[30184]: osdmap e724: 8 total, 8 up, 8 in 2024-03-11T21:07:35.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:35.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:35 smithi129 ceph-mon[42066]: osdmap e724: 8 total, 8 up, 8 in 2024-03-11T21:07:35.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:35.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:35 smithi157 ceph-mon[33359]: osdmap e724: 8 total, 8 up, 8 in 2024-03-11T21:07:36.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:36 smithi129 ceph-mon[30184]: pgmap v1122: 236 pgs: 236 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:07:36.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:36 smithi129 ceph-mon[30184]: osdmap e725: 8 total, 8 up, 8 in 2024-03-11T21:07:36.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:36.474 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:07:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:07:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:07:36.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:36 smithi129 ceph-mon[42066]: pgmap v1122: 236 pgs: 236 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:07:36.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:36 smithi129 ceph-mon[42066]: osdmap e725: 8 total, 8 up, 8 in 2024-03-11T21:07:36.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:36.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:36 smithi157 ceph-mon[33359]: pgmap v1122: 236 pgs: 236 active+clean; 4.4 MiB data, 812 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:07:36.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:36 smithi157 ceph-mon[33359]: osdmap e725: 8 total, 8 up, 8 in 2024-03-11T21:07:36.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:37.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:07:37.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:37 smithi129 ceph-mon[30184]: osdmap e726: 8 total, 8 up, 8 in 2024-03-11T21:07:37.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:07:37.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:07:37.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:37 smithi129 ceph-mon[42066]: osdmap e726: 8 total, 8 up, 8 in 2024-03-11T21:07:37.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:07:37.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:07:37.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:37 smithi157 ceph-mon[33359]: osdmap e726: 8 total, 8 up, 8 in 2024-03-11T21:07:37.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:07:38.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:38 smithi129 ceph-mon[42066]: pgmap v1125: 268 pgs: 268 active+clean; 4.4 MiB data, 821 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:07:38.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:07:38.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:38 smithi129 ceph-mon[42066]: osdmap e727: 8 total, 8 up, 8 in 2024-03-11T21:07:38.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-146"}]: dispatch 2024-03-11T21:07:38.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:38 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-146"}]': finished 2024-03-11T21:07:38.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:38 smithi129 ceph-mon[30184]: pgmap v1125: 268 pgs: 268 active+clean; 4.4 MiB data, 821 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:07:38.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:07:38.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:38 smithi129 ceph-mon[30184]: osdmap e727: 8 total, 8 up, 8 in 2024-03-11T21:07:38.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-146"}]: dispatch 2024-03-11T21:07:38.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:38 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-146"}]': finished 2024-03-11T21:07:38.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:38 smithi157 ceph-mon[33359]: pgmap v1125: 268 pgs: 268 active+clean; 4.4 MiB data, 821 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:07:38.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:07:38.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:38 smithi157 ceph-mon[33359]: osdmap e727: 8 total, 8 up, 8 in 2024-03-11T21:07:38.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-146"}]: dispatch 2024-03-11T21:07:38.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:38 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi129-82818-111", "overlaypool": "test-rados-api-smithi129-82818-146"}]': finished 2024-03-11T21:07:39.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:39 smithi129 ceph-mon[30184]: osdmap e728: 8 total, 8 up, 8 in 2024-03-11T21:07:39.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-146", "mode": "writeback"}]: dispatch 2024-03-11T21:07:39.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:39 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:07:39.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:39 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-146", "mode": "writeback"}]': finished 2024-03-11T21:07:39.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:39 smithi129 ceph-mon[30184]: osdmap e729: 8 total, 8 up, 8 in 2024-03-11T21:07:39.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:39 smithi129 ceph-mon[42066]: osdmap e728: 8 total, 8 up, 8 in 2024-03-11T21:07:39.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-146", "mode": "writeback"}]: dispatch 2024-03-11T21:07:39.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:39 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:07:39.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:39 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-146", "mode": "writeback"}]': finished 2024-03-11T21:07:39.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:39 smithi129 ceph-mon[42066]: osdmap e729: 8 total, 8 up, 8 in 2024-03-11T21:07:39.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:39 smithi157 ceph-mon[33359]: osdmap e728: 8 total, 8 up, 8 in 2024-03-11T21:07:39.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-146", "mode": "writeback"}]: dispatch 2024-03-11T21:07:39.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:39 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-11T21:07:39.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:39 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi129-82818-146", "mode": "writeback"}]': finished 2024-03-11T21:07:39.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:39 smithi157 ceph-mon[33359]: osdmap e729: 8 total, 8 up, 8 in 2024-03-11T21:07:40.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:40 smithi129 ceph-mon[30184]: pgmap v1128: 268 pgs: 268 active+clean; 4.4 MiB data, 821 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:07:40.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:40 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:07:40.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:40 smithi129 ceph-mon[42066]: pgmap v1128: 268 pgs: 268 active+clean; 4.4 MiB data, 821 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:07:40.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:40 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:07:40.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:40 smithi157 ceph-mon[33359]: pgmap v1128: 268 pgs: 268 active+clean; 4.4 MiB data, 821 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-11T21:07:40.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:40 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-11T21:07:41.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:41 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:07:41.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:41 smithi129 ceph-mon[30184]: osdmap e730: 8 total, 8 up, 8 in 2024-03-11T21:07:41.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:41 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:07:41.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:41 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:07:41.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:41 smithi129 ceph-mon[42066]: osdmap e730: 8 total, 8 up, 8 in 2024-03-11T21:07:41.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:41 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:07:41.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:41 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_count","val": "2"}]': finished 2024-03-11T21:07:41.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:41 smithi157 ceph-mon[33359]: osdmap e730: 8 total, 8 up, 8 in 2024-03-11T21:07:41.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:41 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-11T21:07:42.632 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:42 smithi157 ceph-mon[33359]: pgmap v1131: 268 pgs: 268 active+clean; 4.4 MiB data, 821 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:07:42.632 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:07:42.632 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:42 smithi157 ceph-mon[33359]: osdmap e731: 8 total, 8 up, 8 in 2024-03-11T21:07:42.633 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:07:42.633 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:42 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:07:42.633 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:42 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:07:42.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:42 smithi129 ceph-mon[30184]: pgmap v1131: 268 pgs: 268 active+clean; 4.4 MiB data, 821 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:07:42.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:07:42.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:42 smithi129 ceph-mon[30184]: osdmap e731: 8 total, 8 up, 8 in 2024-03-11T21:07:42.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:07:42.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:42 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:07:42.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:42 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:07:42.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:42 smithi129 ceph-mon[42066]: pgmap v1131: 268 pgs: 268 active+clean; 4.4 MiB data, 821 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:07:42.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_period","val": "600"}]': finished 2024-03-11T21:07:42.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:42 smithi129 ceph-mon[42066]: osdmap e731: 8 total, 8 up, 8 in 2024-03-11T21:07:42.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-11T21:07:42.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:42 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-11T21:07:42.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:42 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-11T21:07:43.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:43 smithi129 ceph-mon[30184]: osdmap e732: 8 total, 8 up, 8 in 2024-03-11T21:07:43.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:07:43.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:07:43.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:43 smithi129 ceph-mon[30184]: osdmap e733: 8 total, 8 up, 8 in 2024-03-11T21:07:43.725 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T21:07:43.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:43 smithi129 ceph-mon[42066]: osdmap e732: 8 total, 8 up, 8 in 2024-03-11T21:07:43.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:07:43.725 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:07:43.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:43 smithi129 ceph-mon[42066]: osdmap e733: 8 total, 8 up, 8 in 2024-03-11T21:07:43.726 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T21:07:43.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:43 smithi157 ceph-mon[33359]: osdmap e732: 8 total, 8 up, 8 in 2024-03-11T21:07:43.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-11T21:07:43.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-11T21:07:43.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:43 smithi157 ceph-mon[33359]: osdmap e733: 8 total, 8 up, 8 in 2024-03-11T21:07:43.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-11T21:07:44.306 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:07:43 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:07:44.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:44 smithi129 ceph-mon[30184]: pgmap v1134: 268 pgs: 268 active+clean; 4.4 MiB data, 821 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-11T21:07:44.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "target_max_objects","val": "1"}]': finished 2024-03-11T21:07:44.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:44 smithi129 ceph-mon[30184]: osdmap e734: 8 total, 8 up, 8 in 2024-03-11T21:07:44.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:44 smithi129 ceph-mon[42066]: pgmap v1134: 268 pgs: 268 active+clean; 4.4 MiB data, 821 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-11T21:07:44.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "target_max_objects","val": "1"}]': finished 2024-03-11T21:07:44.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:44 smithi129 ceph-mon[42066]: osdmap e734: 8 total, 8 up, 8 in 2024-03-11T21:07:44.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:44 smithi157 ceph-mon[33359]: pgmap v1134: 268 pgs: 268 active+clean; 4.4 MiB data, 821 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-11T21:07:44.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-146","var": "target_max_objects","val": "1"}]': finished 2024-03-11T21:07:44.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:44 smithi157 ceph-mon[33359]: osdmap e734: 8 total, 8 up, 8 in 2024-03-11T21:07:45.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:45.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:45.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:46.473 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:07:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:07:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:07:46.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:46 smithi129 ceph-mon[30184]: pgmap v1137: 268 pgs: 268 active+clean; 4.4 MiB data, 821 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-11T21:07:46.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:46 smithi129 ceph-mon[30184]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-11T21:07:46.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:46 smithi129 ceph-mon[42066]: pgmap v1137: 268 pgs: 268 active+clean; 4.4 MiB data, 821 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-11T21:07:46.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:46 smithi129 ceph-mon[42066]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-11T21:07:46.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:46 smithi157 ceph-mon[33359]: pgmap v1137: 268 pgs: 268 active+clean; 4.4 MiB data, 821 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-11T21:07:46.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:46 smithi157 ceph-mon[33359]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-11T21:07:48.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:48 smithi129 ceph-mon[30184]: pgmap v1138: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-11T21:07:48.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:48 smithi129 ceph-mon[42066]: pgmap v1138: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-11T21:07:48.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:48 smithi157 ceph-mon[33359]: pgmap v1138: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-11T21:07:50.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:07:50.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:50.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:50 smithi129 ceph-mon[30184]: pgmap v1139: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 759 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:07:50.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:07:50.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:50.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:50 smithi129 ceph-mon[42066]: pgmap v1139: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 759 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:07:50.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:07:50.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:07:50.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:50 smithi157 ceph-mon[33359]: pgmap v1139: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 759 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:07:52.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:52 smithi129 ceph-mon[30184]: pgmap v1140: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-11T21:07:52.473 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:52 smithi129 ceph-mon[42066]: pgmap v1140: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-11T21:07:52.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:52 smithi157 ceph-mon[33359]: pgmap v1140: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-11T21:07:54.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:54 smithi129 ceph-mon[30184]: pgmap v1141: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:07:54.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:54 smithi129 ceph-mon[42066]: pgmap v1141: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:07:54.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:54 smithi157 ceph-mon[33359]: pgmap v1141: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:07:54.481 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:07:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:07:55.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:55.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:55.473 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:55.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:55.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:07:55.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:56.374 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:07:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:07:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:07:56.375 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:56 smithi129 ceph-mon[30184]: pgmap v1142: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 953 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:07:56.375 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:07:56.375 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:56 smithi129 ceph-mon[30184]: osdmap e735: 8 total, 8 up, 8 in 2024-03-11T21:07:56.375 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-146"}]: dispatch 2024-03-11T21:07:56.375 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:56 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:07:56.375 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:56 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:07:56.375 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:56 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:07:56.376 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:56 smithi129 ceph-mon[42066]: pgmap v1142: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 953 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:07:56.376 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:07:56.376 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:56 smithi129 ceph-mon[42066]: osdmap e735: 8 total, 8 up, 8 in 2024-03-11T21:07:56.376 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-146"}]: dispatch 2024-03-11T21:07:56.376 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:56 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:07:56.376 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:56 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:07:56.376 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:56 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:07:56.481 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 21:07:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=sqlstore.transactions t=2024-03-11T21:07:56.317571693Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T21:07:56.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:56 smithi157 ceph-mon[33359]: pgmap v1142: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 953 B/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:07:56.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:07:56.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:56 smithi157 ceph-mon[33359]: osdmap e735: 8 total, 8 up, 8 in 2024-03-11T21:07:56.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-146"}]: dispatch 2024-03-11T21:07:56.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:56 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:07:56.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:56 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:07:56.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:56 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:07:57.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-146"}]': finished 2024-03-11T21:07:57.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:57 smithi129 ceph-mon[30184]: osdmap e736: 8 total, 8 up, 8 in 2024-03-11T21:07:57.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:57.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-146"}]: dispatch 2024-03-11T21:07:57.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:57 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:07:57.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:57 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:07:57.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:57 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:07:57.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-146"}]': finished 2024-03-11T21:07:57.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:57 smithi129 ceph-mon[42066]: osdmap e736: 8 total, 8 up, 8 in 2024-03-11T21:07:57.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:57.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-146"}]: dispatch 2024-03-11T21:07:57.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:57 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:07:57.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:57 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:07:57.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:57 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:07:57.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-146"}]': finished 2024-03-11T21:07:57.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:57 smithi157 ceph-mon[33359]: osdmap e736: 8 total, 8 up, 8 in 2024-03-11T21:07:57.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:07:57.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-146"}]: dispatch 2024-03-11T21:07:57.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:57 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:07:57.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:57 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:07:57.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:57 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:07:58.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:58 smithi129 ceph-mon[30184]: pgmap v1145: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:07:58.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:58 smithi129 ceph-mon[30184]: osdmap e737: 8 total, 8 up, 8 in 2024-03-11T21:07:58.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:58 smithi129 ceph-mon[30184]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:07:58.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:58 smithi129 ceph-mon[30184]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-11T21:07:58.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:58 smithi129 ceph-mon[42066]: pgmap v1145: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:07:58.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:58 smithi129 ceph-mon[42066]: osdmap e737: 8 total, 8 up, 8 in 2024-03-11T21:07:58.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:58 smithi129 ceph-mon[42066]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:07:58.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:58 smithi129 ceph-mon[42066]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-11T21:07:58.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:58 smithi157 ceph-mon[33359]: pgmap v1145: 268 pgs: 268 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-11T21:07:58.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:58 smithi157 ceph-mon[33359]: osdmap e737: 8 total, 8 up, 8 in 2024-03-11T21:07:58.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:58 smithi157 ceph-mon[33359]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:07:58.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:58 smithi157 ceph-mon[33359]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-11T21:07:59.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:59 smithi129 ceph-mon[30184]: osdmap e738: 8 total, 8 up, 8 in 2024-03-11T21:07:59.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:07:59 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:59.473 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:59 smithi129 ceph-mon[42066]: osdmap e738: 8 total, 8 up, 8 in 2024-03-11T21:07:59.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:07:59 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:07:59.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:59 smithi157 ceph-mon[33359]: osdmap e738: 8 total, 8 up, 8 in 2024-03-11T21:07:59.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:07:59 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:08:00.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:00 smithi129 ceph-mon[30184]: pgmap v1148: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T21:08:00.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:00 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:08:00.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:00 smithi129 ceph-mon[30184]: osdmap e739: 8 total, 8 up, 8 in 2024-03-11T21:08:00.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:00 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:08:00.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:00 smithi129 ceph-mon[42066]: pgmap v1148: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T21:08:00.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:00 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:08:00.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:00 smithi129 ceph-mon[42066]: osdmap e739: 8 total, 8 up, 8 in 2024-03-11T21:08:00.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:00 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:08:00.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:00 smithi157 ceph-mon[33359]: pgmap v1148: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-11T21:08:00.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:00 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:08:00.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:00 smithi157 ceph-mon[33359]: osdmap e739: 8 total, 8 up, 8 in 2024-03-11T21:08:00.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:00 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-11T21:08:01.473 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:08:01.473 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:01 smithi129 ceph-mon[42066]: osdmap e740: 8 total, 8 up, 8 in 2024-03-11T21:08:01.473 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:01 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-148"}]: dispatch 2024-03-11T21:08:01.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:08:01.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:01 smithi129 ceph-mon[30184]: osdmap e740: 8 total, 8 up, 8 in 2024-03-11T21:08:01.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:01 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-148"}]: dispatch 2024-03-11T21:08:01.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-11T21:08:01.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:01 smithi157 ceph-mon[33359]: osdmap e740: 8 total, 8 up, 8 in 2024-03-11T21:08:01.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:01 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-148"}]: dispatch 2024-03-11T21:08:02.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:02 smithi129 ceph-mon[30184]: pgmap v1151: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:08:02.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:02 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-148"}]': finished 2024-03-11T21:08:02.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:02 smithi129 ceph-mon[30184]: osdmap e741: 8 total, 8 up, 8 in 2024-03-11T21:08:02.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:02 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:02.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:02 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-148"}]: dispatch 2024-03-11T21:08:02.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:02 smithi129 ceph-mon[42066]: pgmap v1151: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:08:02.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:02 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-148"}]': finished 2024-03-11T21:08:02.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:02 smithi129 ceph-mon[42066]: osdmap e741: 8 total, 8 up, 8 in 2024-03-11T21:08:02.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:02 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:02.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:02 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-148"}]: dispatch 2024-03-11T21:08:02.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:02 smithi157 ceph-mon[33359]: pgmap v1151: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 822 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:08:02.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:02 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-148"}]': finished 2024-03-11T21:08:02.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:02 smithi157 ceph-mon[33359]: osdmap e741: 8 total, 8 up, 8 in 2024-03-11T21:08:02.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:02 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:02.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:02 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-148"}]: dispatch 2024-03-11T21:08:03.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:03 smithi129 ceph-mon[30184]: osdmap e742: 8 total, 8 up, 8 in 2024-03-11T21:08:03.473 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:03 smithi129 ceph-mon[42066]: osdmap e742: 8 total, 8 up, 8 in 2024-03-11T21:08:03.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:03 smithi157 ceph-mon[33359]: osdmap e742: 8 total, 8 up, 8 in 2024-03-11T21:08:04.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:04 smithi129 ceph-mon[30184]: pgmap v1154: 236 pgs: 236 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:08:04.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:04 smithi129 ceph-mon[30184]: osdmap e743: 8 total, 8 up, 8 in 2024-03-11T21:08:04.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:04 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:08:04.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:08:04.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:04.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:04 smithi129 ceph-mon[42066]: pgmap v1154: 236 pgs: 236 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:08:04.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:04 smithi129 ceph-mon[42066]: osdmap e743: 8 total, 8 up, 8 in 2024-03-11T21:08:04.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:04 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:08:04.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:08:04.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:04.481 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:08:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:08:04.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:04 smithi157 ceph-mon[33359]: pgmap v1154: 236 pgs: 236 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:08:04.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:04 smithi157 ceph-mon[33359]: osdmap e743: 8 total, 8 up, 8 in 2024-03-11T21:08:04.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:04 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:08:04.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:08:04.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:05.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:05.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:08:05.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:05 smithi129 ceph-mon[30184]: osdmap e744: 8 total, 8 up, 8 in 2024-03-11T21:08:05.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:05.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:05 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-150"}]: dispatch 2024-03-11T21:08:05.474 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:05 smithi129 ceph-mon[30184]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:08:05.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:05.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:08:05.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:05 smithi129 ceph-mon[42066]: osdmap e744: 8 total, 8 up, 8 in 2024-03-11T21:08:05.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:05.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:05 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-150"}]: dispatch 2024-03-11T21:08:05.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:05 smithi129 ceph-mon[42066]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:08:05.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:05.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:08:05.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:05 smithi157 ceph-mon[33359]: osdmap e744: 8 total, 8 up, 8 in 2024-03-11T21:08:05.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:05.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:05 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-150"}]: dispatch 2024-03-11T21:08:05.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:05 smithi157 ceph-mon[33359]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:08:06.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:06 smithi129 ceph-mon[30184]: pgmap v1157: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:08:06.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:06 smithi129 ceph-mon[30184]: osdmap e745: 8 total, 8 up, 8 in 2024-03-11T21:08:06.473 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:08:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:08:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:08:06.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:06 smithi129 ceph-mon[42066]: pgmap v1157: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:08:06.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:06 smithi129 ceph-mon[42066]: osdmap e745: 8 total, 8 up, 8 in 2024-03-11T21:08:06.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:06 smithi157 ceph-mon[33359]: pgmap v1157: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-11T21:08:06.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:06 smithi157 ceph-mon[33359]: osdmap e745: 8 total, 8 up, 8 in 2024-03-11T21:08:07.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:07 smithi129 ceph-mon[30184]: osdmap e746: 8 total, 8 up, 8 in 2024-03-11T21:08:07.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:08:07.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:07 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:08:07.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:07 smithi129 ceph-mon[30184]: osdmap e747: 8 total, 8 up, 8 in 2024-03-11T21:08:07.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:07 smithi129 ceph-mon[42066]: osdmap e746: 8 total, 8 up, 8 in 2024-03-11T21:08:07.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:08:07.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:07 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:08:07.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:07 smithi129 ceph-mon[42066]: osdmap e747: 8 total, 8 up, 8 in 2024-03-11T21:08:07.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:07 smithi157 ceph-mon[33359]: osdmap e746: 8 total, 8 up, 8 in 2024-03-11T21:08:07.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:08:07.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:07 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:08:07.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:07 smithi157 ceph-mon[33359]: osdmap e747: 8 total, 8 up, 8 in 2024-03-11T21:08:08.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:08 smithi129 ceph-mon[42066]: pgmap v1160: 268 pgs: 32 creating+peering, 236 active+clean; 4.4 MiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:08:08.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:08.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-152"}]: dispatch 2024-03-11T21:08:08.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:08 smithi129 ceph-mon[42066]: osdmap e748: 8 total, 8 up, 8 in 2024-03-11T21:08:08.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:08 smithi129 ceph-mon[30184]: pgmap v1160: 268 pgs: 32 creating+peering, 236 active+clean; 4.4 MiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:08:08.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:08.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-152"}]: dispatch 2024-03-11T21:08:08.724 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:08 smithi129 ceph-mon[30184]: osdmap e748: 8 total, 8 up, 8 in 2024-03-11T21:08:08.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:08 smithi157 ceph-mon[33359]: pgmap v1160: 268 pgs: 32 creating+peering, 236 active+clean; 4.4 MiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:08:08.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:08.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-152"}]: dispatch 2024-03-11T21:08:08.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:08 smithi157 ceph-mon[33359]: osdmap e748: 8 total, 8 up, 8 in 2024-03-11T21:08:10.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:10 smithi129 ceph-mon[30184]: pgmap v1163: 236 pgs: 236 active+clean; 4.4 MiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:08:10.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:10 smithi129 ceph-mon[30184]: osdmap e749: 8 total, 8 up, 8 in 2024-03-11T21:08:10.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:08:10.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:10 smithi129 ceph-mon[42066]: pgmap v1163: 236 pgs: 236 active+clean; 4.4 MiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:08:10.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:10 smithi129 ceph-mon[42066]: osdmap e749: 8 total, 8 up, 8 in 2024-03-11T21:08:10.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:08:10.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:10 smithi157 ceph-mon[33359]: pgmap v1163: 236 pgs: 236 active+clean; 4.4 MiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-11T21:08:10.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:10 smithi157 ceph-mon[33359]: osdmap e749: 8 total, 8 up, 8 in 2024-03-11T21:08:10.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:08:11.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:08:11.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:11 smithi129 ceph-mon[30184]: osdmap e750: 8 total, 8 up, 8 in 2024-03-11T21:08:11.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-111","var": "dedup_tier","val": "test-rados-api-smithi129-82818-154"}]: dispatch 2024-03-11T21:08:11.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:11.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-154"}]: dispatch 2024-03-11T21:08:11.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:08:11.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:11 smithi129 ceph-mon[42066]: osdmap e750: 8 total, 8 up, 8 in 2024-03-11T21:08:11.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-111","var": "dedup_tier","val": "test-rados-api-smithi129-82818-154"}]: dispatch 2024-03-11T21:08:11.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:11.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-154"}]: dispatch 2024-03-11T21:08:11.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi129-82818-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:08:11.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:11 smithi157 ceph-mon[33359]: osdmap e750: 8 total, 8 up, 8 in 2024-03-11T21:08:11.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi129-82818-111","var": "dedup_tier","val": "test-rados-api-smithi129-82818-154"}]: dispatch 2024-03-11T21:08:11.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:11.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi129-82818-111", "tierpool": "test-rados-api-smithi129-82818-154"}]: dispatch 2024-03-11T21:08:12.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:12 smithi157 ceph-mon[33359]: pgmap v1166: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 825 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:08:12.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:12 smithi157 ceph-mon[33359]: osdmap e751: 8 total, 8 up, 8 in 2024-03-11T21:08:12.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:12 smithi129 ceph-mon[30184]: pgmap v1166: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 825 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:08:12.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:12 smithi129 ceph-mon[30184]: osdmap e751: 8 total, 8 up, 8 in 2024-03-11T21:08:12.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:12 smithi129 ceph-mon[42066]: pgmap v1166: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 825 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:08:12.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:12 smithi129 ceph-mon[42066]: osdmap e751: 8 total, 8 up, 8 in 2024-03-11T21:08:13.211 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.212 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.212 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.212 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.212 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.212 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.212 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.212 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.212 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.212 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.212 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.212 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.212 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.213 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.213 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.213 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.213 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.213 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.213 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.213 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.213 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.213 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.213 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.213 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.213 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.213 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.214 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.215 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.216 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.217 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.218 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.219 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.219 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.219 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:08:13.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:13 smithi157 ceph-mon[33359]: osdmap e752: 8 total, 8 up, 8 in 2024-03-11T21:08:13.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:13.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:08:13.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:13 smithi157 ceph-mon[33359]: osdmap e753: 8 total, 8 up, 8 in 2024-03-11T21:08:13.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:13 smithi129 ceph-mon[30184]: osdmap e752: 8 total, 8 up, 8 in 2024-03-11T21:08:13.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:13.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:08:13.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:13 smithi129 ceph-mon[30184]: osdmap e753: 8 total, 8 up, 8 in 2024-03-11T21:08:13.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:13 smithi129 ceph-mon[42066]: osdmap e752: 8 total, 8 up, 8 in 2024-03-11T21:08:13.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:13.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:08:13.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:13 smithi129 ceph-mon[42066]: osdmap e753: 8 total, 8 up, 8 in 2024-03-11T21:08:14.238 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: wait LibRadosTwoPoolsECPP.TryFlush (7947 ms) 2024-03-11T21:08:14.238 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-03-11T21:08:14.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (12987 ms) 2024-03-11T21:08:14.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-03-11T21:08:14.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8057 ms) 2024-03-11T21:08:14.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-03-11T21:08:14.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13099 ms) 2024-03-11T21:08:14.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-03-11T21:08:14.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7834 ms) 2024-03-11T21:08:14.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-03-11T21:08:14.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8035 ms) 2024-03-11T21:08:14.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-03-11T21:08:14.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-11T21:08:14.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: ok, hit_set contains 336:602f83fe:::foo:head 2024-03-11T21:08:14.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9054 ms) 2024-03-11T21:08:14.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-03-11T21:08:14.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710191209,0 2024-03-11T21:08:14.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: first is 1710191209 2024-03-11T21:08:14.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710191209,0 2024-03-11T21:08:14.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710191209,0 2024-03-11T21:08:14.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710191209,1710191211,1710191212,0 2024-03-11T21:08:14.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710191209,1710191211,1710191212,0 2024-03-11T21:08:14.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710191209,1710191211,1710191212,0 2024-03-11T21:08:14.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710191209,1710191211,1710191212,1710191214,1710191215,0 2024-03-11T21:08:14.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710191209,1710191211,1710191212,1710191214,1710191215,0 2024-03-11T21:08:14.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710191209,1710191211,1710191212,1710191214,1710191215,0 2024-03-11T21:08:14.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710191209,1710191211,1710191212,1710191214,1710191215,1710191217,1710191218,0 2024-03-11T21:08:14.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710191209,1710191211,1710191212,1710191214,1710191215,1710191217,1710191218,0 2024-03-11T21:08:14.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710191209,1710191211,1710191212,1710191214,1710191215,1710191217,1710191218,0 2024-03-11T21:08:14.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: got ls 1710191212,1710191214,1710191215,1710191217,1710191218,1710191220,1710191221,0 2024-03-11T21:08:14.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: first now 1710191212, trimmed 2024-03-11T21:08:14.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20920 ms) 2024-03-11T21:08:14.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-03-11T21:08:14.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: foo0 2024-03-11T21:08:14.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-11T21:08:14.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14084 ms) 2024-03-11T21:08:14.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-03-11T21:08:14.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18081 ms) 2024-03-11T21:08:14.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-03-11T21:08:14.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22884 ms) 2024-03-11T21:08:14.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-03-11T21:08:14.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5032 ms) 2024-03-11T21:08:14.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-03-11T21:08:14.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3029 ms) 2024-03-11T21:08:14.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-03-11T21:08:14.242 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3017 ms) 2024-03-11T21:08:14.242 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-03-11T21:08:14.242 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3025 ms) 2024-03-11T21:08:14.242 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (232767 ms total) 2024-03-11T21:08:14.242 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: 2024-03-11T21:08:14.242 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-03-11T21:08:14.242 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (860467 ms total) 2024-03-11T21:08:14.242 INFO:tasks.workunit.client.0.smithi129.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-03-11T21:08:14.243 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:08:14.243 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82522 2024-03-11T21:08:14.243 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82522 2024-03-11T21:08:14.243 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:08:14.244 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82705 2024-03-11T21:08:14.244 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82705 2024-03-11T21:08:14.244 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:08:14.244 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83234 2024-03-11T21:08:14.244 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83234 2024-03-11T21:08:14.244 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:08:14.244 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83020 2024-03-11T21:08:14.244 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83020 2024-03-11T21:08:14.244 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:08:14.244 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82899 2024-03-11T21:08:14.244 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82899 2024-03-11T21:08:14.244 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:08:14.244 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82599 2024-03-11T21:08:14.245 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82599 2024-03-11T21:08:14.480 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:08:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:08:14.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:14 smithi157 ceph-mon[33359]: pgmap v1169: 228 pgs: 228 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:08:14.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:14 smithi157 ceph-mon[33359]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:08:14.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:14.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:08:14.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:14 smithi157 ceph-mon[33359]: osdmap e754: 8 total, 8 up, 8 in 2024-03-11T21:08:14.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:14 smithi129 ceph-mon[30184]: pgmap v1169: 228 pgs: 228 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:08:14.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:14 smithi129 ceph-mon[30184]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:08:14.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:14.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:08:14.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:14 smithi129 ceph-mon[30184]: osdmap e754: 8 total, 8 up, 8 in 2024-03-11T21:08:14.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:14 smithi129 ceph-mon[42066]: pgmap v1169: 228 pgs: 228 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:08:14.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:14 smithi129 ceph-mon[42066]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:08:14.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-111"}]: dispatch 2024-03-11T21:08:14.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1382879453' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi129-82818-111"}]': finished 2024-03-11T21:08:14.724 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:14 smithi129 ceph-mon[42066]: osdmap e754: 8 total, 8 up, 8 in 2024-03-11T21:08:15.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:15.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:15.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:16.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:16 smithi129 ceph-mon[30184]: pgmap v1172: 228 pgs: 228 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:08:16.473 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:08:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:08:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:08:16.474 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:16 smithi129 ceph-mon[42066]: pgmap v1172: 228 pgs: 228 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:08:16.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:16 smithi157 ceph-mon[33359]: pgmap v1172: 228 pgs: 228 active+clean; 582 KiB data, 843 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:08:18.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:18 smithi129 ceph-mon[42066]: pgmap v1173: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:08:18.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:18 smithi129 ceph-mon[30184]: pgmap v1173: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:08:18.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:18 smithi157 ceph-mon[33359]: pgmap v1173: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:08:19.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:19.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:19.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:20.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:20 smithi129 ceph-mon[30184]: pgmap v1174: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 750 B/s rd, 0 op/s 2024-03-11T21:08:20.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:20 smithi129 ceph-mon[42066]: pgmap v1174: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 750 B/s rd, 0 op/s 2024-03-11T21:08:20.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:20 smithi157 ceph-mon[33359]: pgmap v1174: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 750 B/s rd, 0 op/s 2024-03-11T21:08:22.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:22 smithi129 ceph-mon[30184]: pgmap v1175: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:08:22.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:22 smithi129 ceph-mon[42066]: pgmap v1175: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:08:22.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:22 smithi157 ceph-mon[33359]: pgmap v1175: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:08:24.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:24 smithi157 ceph-mon[33359]: pgmap v1176: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:08:24.481 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:08:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:08:24.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:24 smithi129 ceph-mon[30184]: pgmap v1176: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:08:24.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:24 smithi129 ceph-mon[42066]: pgmap v1176: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:08:25.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:25.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:25.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:26.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:26 smithi129 ceph-mon[30184]: pgmap v1177: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 946 B/s rd, 0 op/s 2024-03-11T21:08:26.473 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:08:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:08:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:08:26.473 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:26 smithi129 ceph-mon[42066]: pgmap v1177: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 946 B/s rd, 0 op/s 2024-03-11T21:08:26.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:26 smithi157 ceph-mon[33359]: pgmap v1177: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 946 B/s rd, 0 op/s 2024-03-11T21:08:28.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:28 smithi129 ceph-mon[30184]: pgmap v1178: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:28.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:28 smithi129 ceph-mon[42066]: pgmap v1178: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:28.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:28 smithi157 ceph-mon[33359]: pgmap v1178: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:30.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:30 smithi129 ceph-mon[30184]: pgmap v1179: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:08:30.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:30 smithi129 ceph-mon[42066]: pgmap v1179: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:08:30.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:30 smithi157 ceph-mon[33359]: pgmap v1179: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:08:32.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:32 smithi129 ceph-mon[30184]: pgmap v1180: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:08:32.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:32 smithi129 ceph-mon[42066]: pgmap v1180: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:08:32.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:32 smithi157 ceph-mon[33359]: pgmap v1180: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:08:34.480 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:08:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:08:34.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:34 smithi157 ceph-mon[33359]: pgmap v1181: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:34.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:34 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:34.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:34 smithi129 ceph-mon[30184]: pgmap v1181: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:34.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:34 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:34.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:34 smithi129 ceph-mon[42066]: pgmap v1181: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:34.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:34 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:35.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:35.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:35.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:36.473 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:36 smithi129 ceph-mon[30184]: pgmap v1182: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:08:36.473 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:36 smithi129 ceph-mon[42066]: pgmap v1182: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:08:36.473 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:08:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:08:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:08:36.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:36 smithi157 ceph-mon[33359]: pgmap v1182: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:08:38.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:38 smithi129 ceph-mon[30184]: pgmap v1183: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:38.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:38 smithi129 ceph-mon[42066]: pgmap v1183: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:38.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:38 smithi157 ceph-mon[33359]: pgmap v1183: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:40.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:40 smithi129 ceph-mon[30184]: pgmap v1184: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:08:40.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:40 smithi129 ceph-mon[42066]: pgmap v1184: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:08:40.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:40 smithi157 ceph-mon[33359]: pgmap v1184: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:08:42.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:42 smithi129 ceph-mon[30184]: pgmap v1185: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:08:42.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:42 smithi129 ceph-mon[42066]: pgmap v1185: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:08:42.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:42 smithi157 ceph-mon[33359]: pgmap v1185: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:08:44.299 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:08:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:08:44.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:44 smithi129 ceph-mon[30184]: pgmap v1186: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:44.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:44 smithi129 ceph-mon[42066]: pgmap v1186: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:44.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:44 smithi157 ceph-mon[33359]: pgmap v1186: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:45.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:45.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:45.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:46.472 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:46 smithi129 ceph-mon[30184]: pgmap v1187: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:08:46.473 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:08:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:08:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:08:46.473 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:46 smithi129 ceph-mon[42066]: pgmap v1187: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:08:46.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:46 smithi157 ceph-mon[33359]: pgmap v1187: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:08:48.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:48 smithi129 ceph-mon[30184]: pgmap v1188: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:48.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:48 smithi129 ceph-mon[42066]: pgmap v1188: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:48.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:48 smithi157 ceph-mon[33359]: pgmap v1188: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:49.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:49 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:49.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:49 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:49.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:49 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:08:50.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:50 smithi129 ceph-mon[30184]: pgmap v1189: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:08:50.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:50 smithi129 ceph-mon[42066]: pgmap v1189: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:08:50.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:50 smithi157 ceph-mon[33359]: pgmap v1189: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:08:52.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:52 smithi129 ceph-mon[30184]: pgmap v1190: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:08:52.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:52 smithi129 ceph-mon[42066]: pgmap v1190: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:08:52.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:52 smithi157 ceph-mon[33359]: pgmap v1190: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:08:54.326 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:08:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:08:54.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:54 smithi129 ceph-mon[30184]: pgmap v1191: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:54.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:54 smithi129 ceph-mon[42066]: pgmap v1191: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:54.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:54 smithi157 ceph-mon[33359]: pgmap v1191: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:55.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:55.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:55.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:08:56.472 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:56 smithi129 ceph-mon[30184]: pgmap v1192: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:08:56.473 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:56 smithi129 ceph-mon[42066]: pgmap v1192: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:08:56.473 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:08:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:08:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:08:56.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:56 smithi157 ceph-mon[33359]: pgmap v1192: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:08:57.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:57 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:08:57.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:57 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:08:57.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:57 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:08:57.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:57 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:08:57.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:57 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:08:57.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:57 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:08:57.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:57 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:08:57.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:57 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:08:57.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:57 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:08:57.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:57 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:08:57.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:57 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:08:57.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:57 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:08:58.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:08:58 smithi129 ceph-mon[30184]: pgmap v1193: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:58.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:08:58 smithi129 ceph-mon[42066]: pgmap v1193: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:08:58.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:08:58 smithi157 ceph-mon[33359]: pgmap v1193: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:00.723 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:00 smithi129 ceph-mon[30184]: pgmap v1194: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:00.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:00 smithi129 ceph-mon[42066]: pgmap v1194: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:00.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:00 smithi157 ceph-mon[33359]: pgmap v1194: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:02.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:02 smithi129 ceph-mon[30184]: pgmap v1195: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:02.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:02 smithi129 ceph-mon[42066]: pgmap v1195: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:02.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:02 smithi157 ceph-mon[33359]: pgmap v1195: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:04.346 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:09:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:09:04.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:04 smithi129 ceph-mon[30184]: pgmap v1196: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:04.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:04.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:04 smithi129 ceph-mon[42066]: pgmap v1196: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:04.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:04.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:04 smithi157 ceph-mon[33359]: pgmap v1196: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:04.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:05.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:05.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:05.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:06.352 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:09:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:09:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:09:06.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:06 smithi129 ceph-mon[30184]: pgmap v1197: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:06.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:06 smithi129 ceph-mon[42066]: pgmap v1197: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:06.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:06 smithi157 ceph-mon[33359]: pgmap v1197: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:08.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:08 smithi129 ceph-mon[30184]: pgmap v1198: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:08.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:08 smithi129 ceph-mon[42066]: pgmap v1198: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:08.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:08 smithi157 ceph-mon[33359]: pgmap v1198: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:10.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:10 smithi129 ceph-mon[30184]: pgmap v1199: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:10.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:10 smithi129 ceph-mon[42066]: pgmap v1199: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:10.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:10 smithi157 ceph-mon[33359]: pgmap v1199: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:12.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:12 smithi129 ceph-mon[30184]: pgmap v1200: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:12.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:12 smithi129 ceph-mon[42066]: pgmap v1200: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:12.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:12 smithi157 ceph-mon[33359]: pgmap v1200: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:14.364 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:09:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:09:14.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:14 smithi129 ceph-mon[30184]: pgmap v1201: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:14.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:14 smithi129 ceph-mon[42066]: pgmap v1201: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:14.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:14 smithi157 ceph-mon[33359]: pgmap v1201: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:15.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:15.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:15.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:16.371 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:09:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:09:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:09:16.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:16 smithi129 ceph-mon[30184]: pgmap v1202: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:09:16.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:16 smithi129 ceph-mon[42066]: pgmap v1202: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:09:16.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:16 smithi157 ceph-mon[33359]: pgmap v1202: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:09:18.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:18 smithi129 ceph-mon[30184]: pgmap v1203: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:18.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:18 smithi129 ceph-mon[42066]: pgmap v1203: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:18.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:18 smithi157 ceph-mon[33359]: pgmap v1203: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:19.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:19.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:19.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:20.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:20 smithi129 ceph-mon[30184]: pgmap v1204: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:09:20.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:20 smithi129 ceph-mon[42066]: pgmap v1204: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:09:20.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:20 smithi157 ceph-mon[33359]: pgmap v1204: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:09:22.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:22 smithi129 ceph-mon[30184]: pgmap v1205: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:09:22.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:22 smithi129 ceph-mon[42066]: pgmap v1205: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:09:22.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:22 smithi157 ceph-mon[33359]: pgmap v1205: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:09:24.388 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:09:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:09:24.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:24 smithi129 ceph-mon[30184]: pgmap v1206: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:24.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:24 smithi129 ceph-mon[42066]: pgmap v1206: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:24.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:24 smithi157 ceph-mon[33359]: pgmap v1206: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:25.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:25.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:25.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:26.396 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:09:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:09:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:09:26.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:26 smithi129 ceph-mon[42066]: pgmap v1207: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:26.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:26 smithi129 ceph-mon[30184]: pgmap v1207: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:26.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:26 smithi157 ceph-mon[33359]: pgmap v1207: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:28.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:28 smithi129 ceph-mon[30184]: pgmap v1208: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:28.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:28 smithi129 ceph-mon[42066]: pgmap v1208: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:28.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:28 smithi157 ceph-mon[33359]: pgmap v1208: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:30.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:30 smithi129 ceph-mon[30184]: pgmap v1209: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:09:30.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:30 smithi129 ceph-mon[42066]: pgmap v1209: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:09:30.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:30 smithi157 ceph-mon[33359]: pgmap v1209: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:09:32.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:32 smithi129 ceph-mon[30184]: pgmap v1210: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:32.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:32 smithi129 ceph-mon[42066]: pgmap v1210: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:32.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:32 smithi157 ceph-mon[33359]: pgmap v1210: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:34.410 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:09:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:09:34.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:34 smithi129 ceph-mon[30184]: pgmap v1211: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:34.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:34 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:34.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:34 smithi129 ceph-mon[42066]: pgmap v1211: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:34.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:34 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:34.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:34 smithi157 ceph-mon[33359]: pgmap v1211: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:34.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:34 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:35.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:35.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:35.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:36.419 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:09:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:09:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:09:36.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:36 smithi129 ceph-mon[42066]: pgmap v1212: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:09:36.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:36 smithi129 ceph-mon[30184]: pgmap v1212: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:09:36.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:36 smithi157 ceph-mon[33359]: pgmap v1212: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:09:38.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:38 smithi129 ceph-mon[42066]: pgmap v1213: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:38.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:38 smithi129 ceph-mon[30184]: pgmap v1213: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:38.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:38 smithi157 ceph-mon[33359]: pgmap v1213: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:40.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:40 smithi129 ceph-mon[30184]: pgmap v1214: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:40.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:40 smithi129 ceph-mon[42066]: pgmap v1214: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:40.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:40 smithi157 ceph-mon[33359]: pgmap v1214: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:42.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:42 smithi129 ceph-mon[30184]: pgmap v1215: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:42.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:42 smithi129 ceph-mon[42066]: pgmap v1215: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:42.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:42 smithi157 ceph-mon[33359]: pgmap v1215: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:44.430 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:09:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:09:44.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:44 smithi129 ceph-mon[30184]: pgmap v1216: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:44.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:44 smithi129 ceph-mon[42066]: pgmap v1216: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:44.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:44 smithi157 ceph-mon[33359]: pgmap v1216: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:45.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:45.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:45.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:46.440 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:09:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:09:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:09:46.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:46 smithi129 ceph-mon[30184]: pgmap v1217: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:46.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:46 smithi129 ceph-mon[42066]: pgmap v1217: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:46.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:46 smithi157 ceph-mon[33359]: pgmap v1217: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:48.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:48 smithi129 ceph-mon[30184]: pgmap v1218: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:48.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:48 smithi129 ceph-mon[42066]: pgmap v1218: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:48.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:48 smithi157 ceph-mon[33359]: pgmap v1218: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:49.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:49 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:49.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:49 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:49.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:49 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:09:50.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:50 smithi129 ceph-mon[30184]: pgmap v1219: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:50.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:50 smithi129 ceph-mon[42066]: pgmap v1219: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:50.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:50 smithi157 ceph-mon[33359]: pgmap v1219: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:52.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:52 smithi129 ceph-mon[30184]: pgmap v1220: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:52.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:52 smithi129 ceph-mon[42066]: pgmap v1220: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:52.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:52 smithi157 ceph-mon[33359]: pgmap v1220: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:53.223 INFO:tasks.workunit.client.0.smithi129.stdout:ing 1 sec 2024-03-11T21:09:53.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.223 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.224 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.225 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.226 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.226 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.226 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.226 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.226 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.226 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.226 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.226 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.226 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.226 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.226 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.226 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.226 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.227 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.228 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.229 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:53.230 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:09:54.457 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:09:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:09:54.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:54 smithi129 ceph-mon[30184]: pgmap v1221: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:54.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:54 smithi129 ceph-mon[42066]: pgmap v1221: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:54.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:54 smithi157 ceph-mon[33359]: pgmap v1221: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:55.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:55.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:55.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:09:56.463 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:09:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:09:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:09:56.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:56 smithi129 ceph-mon[30184]: pgmap v1222: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:56.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:56 smithi129 ceph-mon[42066]: pgmap v1222: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:56.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:56 smithi157 ceph-mon[33359]: pgmap v1222: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:09:57.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:57 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:09:57.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:57 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:09:57.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:57 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:09:58.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:58 smithi129 ceph-mon[30184]: pgmap v1223: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:58.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:58 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:09:58.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:58 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:09:58.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:09:58 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:09:58.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:58 smithi129 ceph-mon[42066]: pgmap v1223: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:58.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:58 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:09:58.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:58 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:09:58.723 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:09:58 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:09:58.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:58 smithi157 ceph-mon[33359]: pgmap v1223: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:09:58.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:58 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:09:58.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:58 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:09:58.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:09:58 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:10:00.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:00 smithi157 ceph-mon[33359]: pgmap v1224: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:00.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:00 smithi157 ceph-mon[33359]: overall HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-11T21:10:00.972 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:00 smithi129 ceph-mon[30184]: pgmap v1224: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:00.972 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:00 smithi129 ceph-mon[30184]: overall HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-11T21:10:00.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:00 smithi129 ceph-mon[42066]: pgmap v1224: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:00.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:00 smithi129 ceph-mon[42066]: overall HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-11T21:10:02.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:02 smithi157 ceph-mon[33359]: pgmap v1225: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:02.972 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:02 smithi129 ceph-mon[30184]: pgmap v1225: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:02.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:02 smithi129 ceph-mon[42066]: pgmap v1225: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:04.479 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:10:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:10:04.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:04 smithi157 ceph-mon[33359]: pgmap v1226: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:04.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:04.972 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:04 smithi129 ceph-mon[30184]: pgmap v1226: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:04.972 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:04.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:04 smithi129 ceph-mon[42066]: pgmap v1226: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:04.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:05.722 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:05.722 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:05.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:06.472 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:10:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:10:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:10:06.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:06 smithi129 ceph-mon[30184]: pgmap v1227: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:06.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:06 smithi129 ceph-mon[42066]: pgmap v1227: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:06.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:06 smithi157 ceph-mon[33359]: pgmap v1227: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:08.972 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:08 smithi129 ceph-mon[30184]: pgmap v1228: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:08.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:08 smithi129 ceph-mon[42066]: pgmap v1228: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:08.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:08 smithi157 ceph-mon[33359]: pgmap v1228: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:10.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:10 smithi129 ceph-mon[30184]: pgmap v1229: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:10.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:10 smithi129 ceph-mon[42066]: pgmap v1229: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:10.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:10 smithi157 ceph-mon[33359]: pgmap v1229: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:12.781 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:12 smithi157 ceph-mon[33359]: pgmap v1230: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:12.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:12 smithi129 ceph-mon[30184]: pgmap v1230: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:12.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:12 smithi129 ceph-mon[42066]: pgmap v1230: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:14.479 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:10:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:10:14.972 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:14 smithi129 ceph-mon[30184]: pgmap v1231: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:14.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:14 smithi129 ceph-mon[42066]: pgmap v1231: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:14.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:14 smithi157 ceph-mon[33359]: pgmap v1231: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:15.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:15.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:15.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:16.471 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:10:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:10:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:10:16.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:16 smithi129 ceph-mon[30184]: pgmap v1232: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:16.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:16 smithi129 ceph-mon[42066]: pgmap v1232: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:16.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:16 smithi157 ceph-mon[33359]: pgmap v1232: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:18.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:18 smithi129 ceph-mon[30184]: pgmap v1233: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:18.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:18 smithi129 ceph-mon[42066]: pgmap v1233: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:18.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:18 smithi157 ceph-mon[33359]: pgmap v1233: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:19.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:19.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:19.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:20.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:20 smithi129 ceph-mon[30184]: pgmap v1234: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:20.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:20 smithi129 ceph-mon[42066]: pgmap v1234: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:20.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:20 smithi157 ceph-mon[33359]: pgmap v1234: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:22.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:22 smithi129 ceph-mon[30184]: pgmap v1235: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:22.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:22 smithi129 ceph-mon[42066]: pgmap v1235: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:22.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:22 smithi157 ceph-mon[33359]: pgmap v1235: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:24.479 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:10:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:10:24.972 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:24 smithi129 ceph-mon[30184]: pgmap v1236: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:24.973 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:24 smithi129 ceph-mon[42066]: pgmap v1236: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:24.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:24 smithi157 ceph-mon[33359]: pgmap v1236: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:25.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:25.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:25.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:26.471 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:10:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:10:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:10:26.972 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:26 smithi129 ceph-mon[30184]: pgmap v1237: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:26.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:26 smithi129 ceph-mon[42066]: pgmap v1237: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:26.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:26 smithi157 ceph-mon[33359]: pgmap v1237: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:28.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:28 smithi129 ceph-mon[30184]: pgmap v1238: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:28.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:28 smithi129 ceph-mon[42066]: pgmap v1238: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:28.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:28 smithi157 ceph-mon[33359]: pgmap v1238: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:30.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:30 smithi129 ceph-mon[30184]: pgmap v1239: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:30.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:30 smithi129 ceph-mon[42066]: pgmap v1239: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:30.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:30 smithi157 ceph-mon[33359]: pgmap v1239: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:32.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:32 smithi129 ceph-mon[30184]: pgmap v1240: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:32.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:32 smithi129 ceph-mon[42066]: pgmap v1240: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:32.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:32 smithi157 ceph-mon[33359]: pgmap v1240: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:34.479 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:10:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:10:34.972 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:34 smithi129 ceph-mon[30184]: pgmap v1241: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:34.972 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:34 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:34.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:34 smithi129 ceph-mon[42066]: pgmap v1241: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:34.973 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:34 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:34.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:34 smithi157 ceph-mon[33359]: pgmap v1241: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:34.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:34 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:35.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:35.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:35.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:36.471 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:10:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:10:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:10:36.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:36 smithi129 ceph-mon[30184]: pgmap v1242: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:36.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:36 smithi129 ceph-mon[42066]: pgmap v1242: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:36.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:36 smithi157 ceph-mon[33359]: pgmap v1242: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:38.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:38 smithi129 ceph-mon[30184]: pgmap v1243: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:38.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:38 smithi129 ceph-mon[42066]: pgmap v1243: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:38.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:38 smithi157 ceph-mon[33359]: pgmap v1243: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:40.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:40 smithi129 ceph-mon[30184]: pgmap v1244: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:40.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:40 smithi129 ceph-mon[42066]: pgmap v1244: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:40.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:40 smithi157 ceph-mon[33359]: pgmap v1244: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:42.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:42 smithi129 ceph-mon[30184]: pgmap v1245: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:42.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:42 smithi129 ceph-mon[42066]: pgmap v1245: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:42.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:42 smithi157 ceph-mon[33359]: pgmap v1245: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:44.479 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:10:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:10:44.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:44 smithi129 ceph-mon[30184]: pgmap v1246: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:44.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:44 smithi129 ceph-mon[42066]: pgmap v1246: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:44.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:44 smithi157 ceph-mon[33359]: pgmap v1246: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:45.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:45.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:45.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:46.471 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:10:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:10:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:10:46.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:46 smithi129 ceph-mon[30184]: pgmap v1247: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:46.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:46 smithi129 ceph-mon[42066]: pgmap v1247: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:46 smithi157 ceph-mon[33359]: pgmap v1247: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:48.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:48 smithi129 ceph-mon[30184]: pgmap v1248: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:48.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:48 smithi129 ceph-mon[42066]: pgmap v1248: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:48.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:48 smithi157 ceph-mon[33359]: pgmap v1248: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:49.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:49 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:49.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:49 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:49.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:49 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:10:50.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:50 smithi129 ceph-mon[42066]: pgmap v1249: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:50.972 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:50 smithi129 ceph-mon[30184]: pgmap v1249: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:50.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:50 smithi157 ceph-mon[33359]: pgmap v1249: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:52.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:52 smithi129 ceph-mon[30184]: pgmap v1250: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:52.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:52 smithi129 ceph-mon[42066]: pgmap v1250: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:52.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:52 smithi157 ceph-mon[33359]: pgmap v1250: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:10:54.478 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:10:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:10:54.972 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:54 smithi129 ceph-mon[30184]: pgmap v1251: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:54.973 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:54 smithi129 ceph-mon[42066]: pgmap v1251: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:54.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:54 smithi157 ceph-mon[33359]: pgmap v1251: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:55.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:55.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:55.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:10:56.471 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:10:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:10:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:10:56.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:56 smithi129 ceph-mon[30184]: pgmap v1252: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:56.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:56 smithi129 ceph-mon[42066]: pgmap v1252: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:56.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:56 smithi157 ceph-mon[33359]: pgmap v1252: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:10:58.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:58 smithi129 ceph-mon[30184]: pgmap v1253: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:58.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:58 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:10:58.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:58 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:10:58.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:58 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:10:58.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:10:58 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:10:58.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:58 smithi129 ceph-mon[42066]: pgmap v1253: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:58.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:58 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:10:58.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:58 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:10:58.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:58 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:10:58.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:10:58 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:10:58.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:58 smithi157 ceph-mon[33359]: pgmap v1253: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:10:58.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:58 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:10:58.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:58 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:10:58.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:58 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:10:58.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:10:58 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:11:00.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:00 smithi129 ceph-mon[30184]: pgmap v1254: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:00.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:00 smithi129 ceph-mon[42066]: pgmap v1254: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:00.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:00 smithi157 ceph-mon[33359]: pgmap v1254: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:02.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:02 smithi129 ceph-mon[42066]: pgmap v1255: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:02.972 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:02 smithi129 ceph-mon[30184]: pgmap v1255: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:02.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:02 smithi157 ceph-mon[33359]: pgmap v1255: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:04.478 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:11:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:11:04.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:04 smithi129 ceph-mon[30184]: pgmap v1256: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:04.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:04.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:04 smithi129 ceph-mon[42066]: pgmap v1256: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:04.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:04.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:04 smithi157 ceph-mon[33359]: pgmap v1256: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:04.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:05.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:05.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:05.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:06.471 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:11:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:11:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:11:06.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:06 smithi129 ceph-mon[30184]: pgmap v1257: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:06.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:06 smithi129 ceph-mon[42066]: pgmap v1257: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:06.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:06 smithi157 ceph-mon[33359]: pgmap v1257: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:08.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:08 smithi129 ceph-mon[30184]: pgmap v1258: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:08.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:08 smithi129 ceph-mon[42066]: pgmap v1258: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:08.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:08 smithi157 ceph-mon[33359]: pgmap v1258: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:10.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:10 smithi129 ceph-mon[30184]: pgmap v1259: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:10.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:10 smithi129 ceph-mon[42066]: pgmap v1259: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:10.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:10 smithi157 ceph-mon[33359]: pgmap v1259: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:12.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:12 smithi129 ceph-mon[30184]: pgmap v1260: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:12.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:12 smithi129 ceph-mon[42066]: pgmap v1260: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:12.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:12 smithi157 ceph-mon[33359]: pgmap v1260: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:14.478 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:11:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:11:14.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:14 smithi129 ceph-mon[30184]: pgmap v1261: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:14.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:14 smithi129 ceph-mon[42066]: pgmap v1261: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:14.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:14 smithi157 ceph-mon[33359]: pgmap v1261: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:15.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:15.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:15.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:16.471 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:11:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:11:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:11:16.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:16 smithi129 ceph-mon[30184]: pgmap v1262: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:16.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:16 smithi129 ceph-mon[42066]: pgmap v1262: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:16.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:16 smithi157 ceph-mon[33359]: pgmap v1262: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:18.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:18 smithi129 ceph-mon[30184]: pgmap v1263: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:18.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:18 smithi129 ceph-mon[42066]: pgmap v1263: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:18.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:18 smithi157 ceph-mon[33359]: pgmap v1263: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:19.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:19.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:19.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:20.974 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:20 smithi129 ceph-mon[30184]: pgmap v1264: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:20.974 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:20 smithi129 ceph-mon[42066]: pgmap v1264: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:20.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:20 smithi157 ceph-mon[33359]: pgmap v1264: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:22.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:22 smithi129 ceph-mon[30184]: pgmap v1265: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:22.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:22 smithi129 ceph-mon[42066]: pgmap v1265: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:22.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:22 smithi157 ceph-mon[33359]: pgmap v1265: 228 pgs: 228 active+clean; 582 KiB data, 855 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:24.478 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:11:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:11:24.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:24 smithi129 ceph-mon[30184]: pgmap v1266: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:24.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:24 smithi129 ceph-mon[42066]: pgmap v1266: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:24.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:24 smithi157 ceph-mon[33359]: pgmap v1266: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:25.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:25.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:25.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:26.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:11:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:11:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:11:26.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:26 smithi129 ceph-mon[30184]: pgmap v1267: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:26.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:26 smithi129 ceph-mon[42066]: pgmap v1267: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:26.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:26 smithi157 ceph-mon[33359]: pgmap v1267: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:28.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:28 smithi129 ceph-mon[30184]: pgmap v1268: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:28.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:28 smithi129 ceph-mon[42066]: pgmap v1268: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:28.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:28 smithi157 ceph-mon[33359]: pgmap v1268: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:30.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:30 smithi129 ceph-mon[42066]: pgmap v1269: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:30.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:30 smithi129 ceph-mon[30184]: pgmap v1269: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:30.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:30 smithi157 ceph-mon[33359]: pgmap v1269: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:32.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:32 smithi129 ceph-mon[30184]: pgmap v1270: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:32.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:32 smithi129 ceph-mon[42066]: pgmap v1270: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:32.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:32 smithi157 ceph-mon[33359]: pgmap v1270: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:33.233 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.233 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.233 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.234 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.234 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.234 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.234 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.234 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.234 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.234 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.234 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.234 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.234 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.234 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.235 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.235 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.235 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.235 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.235 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.235 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.235 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.235 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.235 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.235 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.235 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.235 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.236 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.236 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.236 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.236 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.236 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.236 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.236 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.236 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.236 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.236 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.236 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.236 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.236 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.237 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.237 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.237 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.237 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.237 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.237 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.237 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.237 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.237 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.237 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.237 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.237 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.237 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.237 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.238 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.238 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.238 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.238 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.238 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.238 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.238 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.238 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.238 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.238 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.238 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.238 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.238 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.239 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.240 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:33.241 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:11:34.478 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:11:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:11:34.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:34 smithi129 ceph-mon[30184]: pgmap v1271: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:34.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:34 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:34.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:34 smithi129 ceph-mon[42066]: pgmap v1271: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:34.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:34 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:34.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:34 smithi157 ceph-mon[33359]: pgmap v1271: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:11:34.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:34 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:35.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:35.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:35.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:36.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:11:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:11:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:11:36.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:36 smithi129 ceph-mon[30184]: pgmap v1272: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:36.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:36 smithi129 ceph-mon[42066]: pgmap v1272: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:36.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:36 smithi157 ceph-mon[33359]: pgmap v1272: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:38.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:38 smithi129 ceph-mon[30184]: pgmap v1273: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:38.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:38 smithi129 ceph-mon[42066]: pgmap v1273: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:38.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:38 smithi157 ceph-mon[33359]: pgmap v1273: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:40.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:40 smithi129 ceph-mon[30184]: pgmap v1274: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:40.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:40 smithi129 ceph-mon[42066]: pgmap v1274: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:40.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:40 smithi157 ceph-mon[33359]: pgmap v1274: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:42.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:42 smithi129 ceph-mon[30184]: pgmap v1275: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:42.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:42 smithi129 ceph-mon[42066]: pgmap v1275: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:42.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:42 smithi157 ceph-mon[33359]: pgmap v1275: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:44.478 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:11:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:11:44.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:44 smithi129 ceph-mon[30184]: pgmap v1276: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:44.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:44 smithi129 ceph-mon[42066]: pgmap v1276: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:44.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:44 smithi157 ceph-mon[33359]: pgmap v1276: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:45.970 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:45.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:45.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:46.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:11:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:11:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:11:46.970 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:46 smithi129 ceph-mon[30184]: pgmap v1277: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:46.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:46 smithi129 ceph-mon[42066]: pgmap v1277: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:46 smithi157 ceph-mon[33359]: pgmap v1277: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:48.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:48 smithi129 ceph-mon[30184]: pgmap v1278: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:48.972 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:48 smithi129 ceph-mon[42066]: pgmap v1278: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:48.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:48 smithi157 ceph-mon[33359]: pgmap v1278: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:49.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:49 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:49.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:49 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:49.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:49 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:11:50.970 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:50 smithi129 ceph-mon[30184]: pgmap v1279: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:50.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:50 smithi129 ceph-mon[42066]: pgmap v1279: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:50.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:50 smithi157 ceph-mon[33359]: pgmap v1279: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:52.970 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:52 smithi129 ceph-mon[30184]: pgmap v1280: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:52.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:52 smithi129 ceph-mon[42066]: pgmap v1280: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:52.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:52 smithi157 ceph-mon[33359]: pgmap v1280: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:11:54.478 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:11:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:11:54.970 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:54 smithi129 ceph-mon[30184]: pgmap v1281: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:54.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:54 smithi129 ceph-mon[42066]: pgmap v1281: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:54.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:54 smithi157 ceph-mon[33359]: pgmap v1281: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:55.971 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:55.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:55.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:11:56.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:11:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:11:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:11:56.970 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:56 smithi129 ceph-mon[30184]: pgmap v1282: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:56.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:56 smithi129 ceph-mon[42066]: pgmap v1282: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:56.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:56 smithi157 ceph-mon[33359]: pgmap v1282: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:11:58.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:58 smithi157 ceph-mon[33359]: pgmap v1283: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:58.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:58 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:11:59.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:58 smithi129 ceph-mon[30184]: pgmap v1283: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:59.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:58 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:11:59.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:58 smithi129 ceph-mon[42066]: pgmap v1283: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:11:59.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:58 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:12:00.221 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:59 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:12:00.221 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:59 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:12:00.221 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:11:59 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:12:00.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:59 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:12:00.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:59 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:12:00.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:11:59 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:12:00.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:59 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:12:00.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:59 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:12:00.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:11:59 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:12:01.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:00 smithi129 ceph-mon[30184]: pgmap v1284: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:01.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:00 smithi129 ceph-mon[42066]: pgmap v1284: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:01.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:00 smithi157 ceph-mon[33359]: pgmap v1284: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:03.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:02 smithi129 ceph-mon[30184]: pgmap v1285: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:03.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:02 smithi129 ceph-mon[42066]: pgmap v1285: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:03.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:02 smithi157 ceph-mon[33359]: pgmap v1285: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:04.478 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:12:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:12:05.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:04 smithi129 ceph-mon[42066]: pgmap v1286: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:05.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:05.221 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:04 smithi129 ceph-mon[30184]: pgmap v1286: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:05.221 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:05.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:04 smithi157 ceph-mon[33359]: pgmap v1286: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:05.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:06.097 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:06.097 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:06.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:06.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:12:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:12:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:12:07.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:06 smithi129 ceph-mon[30184]: pgmap v1287: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:07.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:06 smithi129 ceph-mon[42066]: pgmap v1287: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:07.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:06 smithi157 ceph-mon[33359]: pgmap v1287: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:09.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:08 smithi129 ceph-mon[30184]: pgmap v1288: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:09.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:08 smithi129 ceph-mon[42066]: pgmap v1288: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:09.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:08 smithi157 ceph-mon[33359]: pgmap v1288: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:11.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:10 smithi129 ceph-mon[30184]: pgmap v1289: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:11.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:10 smithi129 ceph-mon[42066]: pgmap v1289: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:11.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:10 smithi157 ceph-mon[33359]: pgmap v1289: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:13.029 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:12 smithi157 ceph-mon[33359]: pgmap v1290: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:13.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:12 smithi129 ceph-mon[30184]: pgmap v1290: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:13.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:12 smithi129 ceph-mon[42066]: pgmap v1290: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:14.478 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:12:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:12:15.221 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:14 smithi129 ceph-mon[30184]: pgmap v1291: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:15.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:14 smithi129 ceph-mon[42066]: pgmap v1291: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:15.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:14 smithi157 ceph-mon[33359]: pgmap v1291: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:16.097 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:16.097 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:16.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:16.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:12:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:12:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:12:17.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:16 smithi129 ceph-mon[30184]: pgmap v1292: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:17.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:16 smithi129 ceph-mon[42066]: pgmap v1292: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:17.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:16 smithi157 ceph-mon[33359]: pgmap v1292: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:19.221 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:18 smithi129 ceph-mon[30184]: pgmap v1293: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:19.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:18 smithi129 ceph-mon[42066]: pgmap v1293: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:19.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:18 smithi157 ceph-mon[33359]: pgmap v1293: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:20.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:20.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:20.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:21.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:20 smithi129 ceph-mon[30184]: pgmap v1294: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:21.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:20 smithi129 ceph-mon[42066]: pgmap v1294: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:21.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:20 smithi157 ceph-mon[33359]: pgmap v1294: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:23.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:22 smithi129 ceph-mon[30184]: pgmap v1295: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:23.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:22 smithi129 ceph-mon[42066]: pgmap v1295: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:23.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:22 smithi157 ceph-mon[33359]: pgmap v1295: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:24.728 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:12:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:12:25.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:24 smithi129 ceph-mon[30184]: pgmap v1296: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:25.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:24 smithi129 ceph-mon[42066]: pgmap v1296: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:25.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:24 smithi157 ceph-mon[33359]: pgmap v1296: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:26.096 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:26.097 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:26.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:26.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:12:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:12:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:12:27.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:26 smithi129 ceph-mon[30184]: pgmap v1297: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:27.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:26 smithi129 ceph-mon[42066]: pgmap v1297: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:27.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:26 smithi157 ceph-mon[33359]: pgmap v1297: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:29.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:28 smithi129 ceph-mon[30184]: pgmap v1298: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:29.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:28 smithi129 ceph-mon[42066]: pgmap v1298: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:29.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:28 smithi157 ceph-mon[33359]: pgmap v1298: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:31.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:30 smithi129 ceph-mon[30184]: pgmap v1299: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:31.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:30 smithi129 ceph-mon[42066]: pgmap v1299: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:31.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:30 smithi157 ceph-mon[33359]: pgmap v1299: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:33.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:32 smithi129 ceph-mon[30184]: pgmap v1300: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:33.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:32 smithi129 ceph-mon[42066]: pgmap v1300: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:33.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:32 smithi157 ceph-mon[33359]: pgmap v1300: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:34.727 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:12:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:12:35.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:34 smithi129 ceph-mon[30184]: pgmap v1301: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:35.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:34 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:35.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:34 smithi129 ceph-mon[42066]: pgmap v1301: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:35.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:34 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:35.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:34 smithi157 ceph-mon[33359]: pgmap v1301: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:35.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:34 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:36.096 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:36.097 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:36.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:36.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:12:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:12:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:12:37.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:36 smithi129 ceph-mon[30184]: pgmap v1302: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:37.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:36 smithi129 ceph-mon[42066]: pgmap v1302: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:37.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:36 smithi157 ceph-mon[33359]: pgmap v1302: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:39.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:38 smithi129 ceph-mon[30184]: pgmap v1303: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:39.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:38 smithi129 ceph-mon[42066]: pgmap v1303: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:39.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:38 smithi157 ceph-mon[33359]: pgmap v1303: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:41.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:40 smithi129 ceph-mon[30184]: pgmap v1304: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:41.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:40 smithi129 ceph-mon[42066]: pgmap v1304: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:41.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:40 smithi157 ceph-mon[33359]: pgmap v1304: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:43.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:42 smithi129 ceph-mon[30184]: pgmap v1305: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:43.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:42 smithi129 ceph-mon[42066]: pgmap v1305: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:43.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:42 smithi157 ceph-mon[33359]: pgmap v1305: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:44.727 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:12:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:12:45.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:44 smithi129 ceph-mon[30184]: pgmap v1306: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:45.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:44 smithi129 ceph-mon[42066]: pgmap v1306: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:45.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:44 smithi157 ceph-mon[33359]: pgmap v1306: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:46.096 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:46.097 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:46.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:46.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:12:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:12:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:12:47.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:46 smithi129 ceph-mon[30184]: pgmap v1307: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:47.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:46 smithi129 ceph-mon[42066]: pgmap v1307: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:47.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:46 smithi157 ceph-mon[33359]: pgmap v1307: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:49.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:48 smithi129 ceph-mon[30184]: pgmap v1308: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:49.221 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:48 smithi129 ceph-mon[42066]: pgmap v1308: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:49.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:48 smithi157 ceph-mon[33359]: pgmap v1308: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:50.222 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:49 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:50.223 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:49 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:50.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:49 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:12:51.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:50 smithi129 ceph-mon[30184]: pgmap v1309: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:51.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:50 smithi129 ceph-mon[42066]: pgmap v1309: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:51.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:50 smithi157 ceph-mon[33359]: pgmap v1309: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:53.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:52 smithi129 ceph-mon[30184]: pgmap v1310: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:53.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:52 smithi129 ceph-mon[42066]: pgmap v1310: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:53.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:52 smithi157 ceph-mon[33359]: pgmap v1310: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:12:54.727 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:12:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:12:55.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:54 smithi129 ceph-mon[30184]: pgmap v1311: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:55.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:54 smithi129 ceph-mon[42066]: pgmap v1311: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:55.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:54 smithi157 ceph-mon[33359]: pgmap v1311: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:56.121 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:56.122 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:56.122 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:12:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:12:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:12:56.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:12:57.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:56 smithi129 ceph-mon[30184]: pgmap v1312: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:57.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:56 smithi129 ceph-mon[42066]: pgmap v1312: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:57.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:56 smithi157 ceph-mon[33359]: pgmap v1312: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:12:57.227 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 21:12:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=cleanup t=2024-03-11T21:12:56.952188449Z level=info msg="Completed cleanup jobs" duration=5.953488ms 2024-03-11T21:12:59.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:58 smithi129 ceph-mon[30184]: pgmap v1313: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:59.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:58 smithi129 ceph-mon[42066]: pgmap v1313: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:12:59.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:58 smithi157 ceph-mon[33359]: pgmap v1313: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:00.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:12:59 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:13:00.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:12:59 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:13:00.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:12:59 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:13:01.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:00 smithi129 ceph-mon[42066]: pgmap v1314: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:01.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:00 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:13:01.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:00 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:13:01.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:00 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:13:01.221 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:00 smithi129 ceph-mon[30184]: pgmap v1314: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:01.221 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:00 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:13:01.221 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:00 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:13:01.221 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:00 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:13:01.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:00 smithi157 ceph-mon[33359]: pgmap v1314: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:01.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:00 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:13:01.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:00 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:13:01.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:00 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:13:03.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:02 smithi129 ceph-mon[30184]: pgmap v1315: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:03.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:02 smithi129 ceph-mon[42066]: pgmap v1315: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:03.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:02 smithi157 ceph-mon[33359]: pgmap v1315: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:04.727 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:13:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:13:05.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:04 smithi129 ceph-mon[30184]: pgmap v1316: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:05.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:05.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:04 smithi129 ceph-mon[42066]: pgmap v1316: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:05.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:05.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:04 smithi157 ceph-mon[33359]: pgmap v1316: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:05.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:06.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:06.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:06.221 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:13:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:13:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:13:06.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:07.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:06 smithi129 ceph-mon[30184]: pgmap v1317: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:07.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:06 smithi129 ceph-mon[42066]: pgmap v1317: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:07.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:06 smithi157 ceph-mon[33359]: pgmap v1317: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:09.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:08 smithi129 ceph-mon[30184]: pgmap v1318: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:09.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:08 smithi129 ceph-mon[42066]: pgmap v1318: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:09.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:08 smithi157 ceph-mon[33359]: pgmap v1318: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:11.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:10 smithi129 ceph-mon[30184]: pgmap v1319: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:11.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:10 smithi129 ceph-mon[42066]: pgmap v1319: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:11.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:10 smithi157 ceph-mon[33359]: pgmap v1319: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:12.220 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:11 smithi129 ceph-mon[30184]: pgmap v1320: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:12.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:11 smithi129 ceph-mon[42066]: pgmap v1320: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:12.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:11 smithi157 ceph-mon[33359]: pgmap v1320: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:13.244 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.244 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.244 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.244 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.244 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.244 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.244 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.244 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.245 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.245 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.245 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.245 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.245 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.245 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.245 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.245 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.245 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.245 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.245 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.245 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.245 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.246 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.246 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.246 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.246 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.246 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.246 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.246 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.246 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.246 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.246 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.246 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.246 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.246 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.246 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.247 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.247 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.247 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.247 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.247 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.247 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.247 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.247 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.247 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.247 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.247 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.247 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.247 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.247 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.248 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.248 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.248 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.248 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.248 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.248 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.248 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.248 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.248 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.248 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.248 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.248 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.248 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.249 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.250 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:13.251 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:13:14.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:14 smithi129 ceph-mon[30184]: pgmap v1321: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:14.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:14 smithi129 ceph-mon[42066]: pgmap v1321: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:14.477 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:13:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:13:14.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:14 smithi157 ceph-mon[33359]: pgmap v1321: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:15.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:15.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:15.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:16.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:16 smithi129 ceph-mon[30184]: pgmap v1322: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:16.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:13:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:13:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:13:16.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:16 smithi129 ceph-mon[42066]: pgmap v1322: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:16.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:16 smithi157 ceph-mon[33359]: pgmap v1322: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:18.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:18 smithi129 ceph-mon[30184]: pgmap v1323: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:18.471 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:18 smithi129 ceph-mon[42066]: pgmap v1323: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:18.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:18 smithi157 ceph-mon[33359]: pgmap v1323: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:20.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:20 smithi129 ceph-mon[30184]: pgmap v1324: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:20.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:20.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:20 smithi129 ceph-mon[42066]: pgmap v1324: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:20.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:20.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:20 smithi157 ceph-mon[33359]: pgmap v1324: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:20.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:22.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:22 smithi129 ceph-mon[30184]: pgmap v1325: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:22.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:22 smithi129 ceph-mon[42066]: pgmap v1325: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:22.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:22 smithi157 ceph-mon[33359]: pgmap v1325: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:24.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:24 smithi157 ceph-mon[33359]: pgmap v1326: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:24.477 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:13:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:13:24.720 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:24 smithi129 ceph-mon[30184]: pgmap v1326: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:24.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:24 smithi129 ceph-mon[42066]: pgmap v1326: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:25.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:25.720 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:25.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:26.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:26 smithi129 ceph-mon[30184]: pgmap v1327: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:26.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:13:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:13:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:13:26.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:26 smithi129 ceph-mon[42066]: pgmap v1327: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:26.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:26 smithi157 ceph-mon[33359]: pgmap v1327: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:28.720 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:28 smithi129 ceph-mon[30184]: pgmap v1328: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:28.721 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:28 smithi129 ceph-mon[42066]: pgmap v1328: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:28.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:28 smithi157 ceph-mon[33359]: pgmap v1328: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:30.720 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:30 smithi129 ceph-mon[30184]: pgmap v1329: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:30.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:30 smithi129 ceph-mon[42066]: pgmap v1329: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:30.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:30 smithi157 ceph-mon[33359]: pgmap v1329: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:32.720 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:32 smithi129 ceph-mon[30184]: pgmap v1330: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:32.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:32 smithi129 ceph-mon[42066]: pgmap v1330: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:32.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:32 smithi157 ceph-mon[33359]: pgmap v1330: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:34.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:34 smithi129 ceph-mon[30184]: pgmap v1331: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:34.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:34 smithi129 ceph-mon[42066]: pgmap v1331: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:34.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:34 smithi157 ceph-mon[33359]: pgmap v1331: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:34.727 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:13:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:13:35.720 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:35.720 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:35 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:35.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:35.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:35 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:35.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:35.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:35 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:36.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:36 smithi129 ceph-mon[30184]: pgmap v1332: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:36.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:13:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:13:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:13:36.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:36 smithi129 ceph-mon[42066]: pgmap v1332: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:36.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:36 smithi157 ceph-mon[33359]: pgmap v1332: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:38.720 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:38 smithi129 ceph-mon[30184]: pgmap v1333: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:38.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:38 smithi129 ceph-mon[42066]: pgmap v1333: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:38.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:38 smithi157 ceph-mon[33359]: pgmap v1333: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:40.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:40 smithi129 ceph-mon[30184]: pgmap v1334: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:40.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:40 smithi129 ceph-mon[42066]: pgmap v1334: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:40.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:40 smithi157 ceph-mon[33359]: pgmap v1334: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:42.720 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:42 smithi129 ceph-mon[30184]: pgmap v1335: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:42.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:42 smithi129 ceph-mon[42066]: pgmap v1335: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:42.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:42 smithi157 ceph-mon[33359]: pgmap v1335: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:13:44.529 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:13:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:13:44.529 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:44 smithi157 ceph-mon[33359]: pgmap v1336: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:44.720 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:44 smithi129 ceph-mon[30184]: pgmap v1336: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:44.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:44 smithi129 ceph-mon[42066]: pgmap v1336: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:45.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:45.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:45.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:46.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:46 smithi129 ceph-mon[30184]: pgmap v1337: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:46.472 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:46 smithi129 ceph-mon[42066]: pgmap v1337: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:46.472 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:13:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:13:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:13:46.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:46 smithi157 ceph-mon[33359]: pgmap v1337: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:48.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:48 smithi129 ceph-mon[30184]: pgmap v1338: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:48.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:48 smithi129 ceph-mon[42066]: pgmap v1338: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:48.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:48 smithi157 ceph-mon[33359]: pgmap v1338: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:50.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:50 smithi129 ceph-mon[30184]: pgmap v1339: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:50.720 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:50.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:50 smithi129 ceph-mon[42066]: pgmap v1339: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:50.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:50.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:50 smithi157 ceph-mon[33359]: pgmap v1339: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:50.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:13:52.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:52 smithi129 ceph-mon[30184]: pgmap v1340: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:52.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:52 smithi129 ceph-mon[42066]: pgmap v1340: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:52.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:52 smithi157 ceph-mon[33359]: pgmap v1340: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:54.720 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:54 smithi129 ceph-mon[30184]: pgmap v1341: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:54.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:54 smithi129 ceph-mon[42066]: pgmap v1341: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:54.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:13:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:13:54.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:54 smithi157 ceph-mon[33359]: pgmap v1341: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:55.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:55.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:55.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:13:56.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:56 smithi129 ceph-mon[30184]: pgmap v1342: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:56.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:56 smithi129 ceph-mon[42066]: pgmap v1342: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:56.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:13:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:13:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:13:56.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:56 smithi157 ceph-mon[33359]: pgmap v1342: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:13:58.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:13:58 smithi129 ceph-mon[30184]: pgmap v1343: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:58.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:13:58 smithi129 ceph-mon[42066]: pgmap v1343: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:13:58.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:13:58 smithi157 ceph-mon[33359]: pgmap v1343: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:00.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:00 smithi129 ceph-mon[30184]: pgmap v1344: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:00.720 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:00 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:14:00.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:00 smithi129 ceph-mon[42066]: pgmap v1344: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:00.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:00 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:14:00.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:00 smithi157 ceph-mon[33359]: pgmap v1344: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:00.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:00 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:14:02.584 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:02 smithi157 ceph-mon[33359]: pgmap v1345: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:02.720 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:02 smithi129 ceph-mon[30184]: pgmap v1345: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:02.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:02 smithi129 ceph-mon[42066]: pgmap v1345: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:04.314 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:14:04.314 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:14:04.314 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:04 smithi157 ceph-mon[33359]: pgmap v1346: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:04.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:14:04.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:14:04.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:04 smithi129 ceph-mon[30184]: pgmap v1346: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:04.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:14:04.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:14:04.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:04 smithi129 ceph-mon[42066]: pgmap v1346: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:04.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:14:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:14:05.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:05.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:05.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:14:05.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:14:05.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:14:05.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:14:05.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:14:05.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:05.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:05.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:14:05.471 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:14:05.471 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:14:05.471 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:14:05.471 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:14:05.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:05.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:05.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:14:05.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:14:05.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:14:05.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:14:05.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:14:06.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:06 smithi129 ceph-mon[30184]: pgmap v1347: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:06.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:14:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:14:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:14:06.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:06 smithi129 ceph-mon[42066]: pgmap v1347: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:06.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:06 smithi157 ceph-mon[33359]: pgmap v1347: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:08.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:08 smithi129 ceph-mon[30184]: pgmap v1348: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:08.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:08 smithi129 ceph-mon[42066]: pgmap v1348: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:08.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:08 smithi157 ceph-mon[33359]: pgmap v1348: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:10.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:10 smithi129 ceph-mon[30184]: pgmap v1349: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:10.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:10 smithi129 ceph-mon[42066]: pgmap v1349: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:10.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:10 smithi157 ceph-mon[33359]: pgmap v1349: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:12.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:12 smithi129 ceph-mon[30184]: pgmap v1350: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:12.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:12 smithi129 ceph-mon[42066]: pgmap v1350: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:12.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:12 smithi157 ceph-mon[33359]: pgmap v1350: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:14.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:14 smithi129 ceph-mon[30184]: pgmap v1351: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:14.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:14 smithi129 ceph-mon[42066]: pgmap v1351: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:14.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:14 smithi157 ceph-mon[33359]: pgmap v1351: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:14.477 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:14:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:14:15.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:15.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:15.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:16.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:16 smithi129 ceph-mon[30184]: pgmap v1352: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:16.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:14:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:14:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:14:16.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:16 smithi129 ceph-mon[42066]: pgmap v1352: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:16.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:16 smithi157 ceph-mon[33359]: pgmap v1352: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:18.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:18 smithi129 ceph-mon[30184]: pgmap v1353: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:18.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:18 smithi129 ceph-mon[42066]: pgmap v1353: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:18.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:18 smithi157 ceph-mon[33359]: pgmap v1353: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:20.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:20 smithi129 ceph-mon[30184]: pgmap v1354: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:20.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:20.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:20 smithi129 ceph-mon[42066]: pgmap v1354: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:20.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:20.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:20 smithi157 ceph-mon[33359]: pgmap v1354: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:20.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:22.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:22 smithi129 ceph-mon[30184]: pgmap v1355: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:22.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:22 smithi129 ceph-mon[42066]: pgmap v1355: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:22.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:22 smithi157 ceph-mon[33359]: pgmap v1355: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:24.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:24 smithi129 ceph-mon[30184]: pgmap v1356: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:24.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:24 smithi129 ceph-mon[42066]: pgmap v1356: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:24.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:24 smithi157 ceph-mon[33359]: pgmap v1356: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:24.477 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:14:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:14:25.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:25.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:25.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:26.371 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:26 smithi129 ceph-mon[30184]: pgmap v1357: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:26.371 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:14:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:14:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:14:26.371 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:26 smithi129 ceph-mon[42066]: pgmap v1357: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:26.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:26 smithi157 ceph-mon[33359]: pgmap v1357: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:28.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:28 smithi129 ceph-mon[30184]: pgmap v1358: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:28.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:28 smithi129 ceph-mon[42066]: pgmap v1358: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:28.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:28 smithi157 ceph-mon[33359]: pgmap v1358: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:30.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:30 smithi129 ceph-mon[30184]: pgmap v1359: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:30.469 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:30 smithi129 ceph-mon[42066]: pgmap v1359: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:30.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:30 smithi157 ceph-mon[33359]: pgmap v1359: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:32.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:32 smithi129 ceph-mon[30184]: pgmap v1360: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:32.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:32 smithi129 ceph-mon[42066]: pgmap v1360: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:32.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:32 smithi157 ceph-mon[33359]: pgmap v1360: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:34.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:34 smithi157 ceph-mon[33359]: pgmap v1361: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:34.477 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:14:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:14:34.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:34 smithi129 ceph-mon[30184]: pgmap v1361: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:34.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:34 smithi129 ceph-mon[42066]: pgmap v1361: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:35.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:35.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:35 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:35.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:35.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:35 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:35.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:35.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:35 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:36.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:36 smithi129 ceph-mon[30184]: pgmap v1362: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:36.469 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:36 smithi129 ceph-mon[42066]: pgmap v1362: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:36.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:14:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:14:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:14:36.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:36 smithi157 ceph-mon[33359]: pgmap v1362: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:38.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:38 smithi129 ceph-mon[42066]: pgmap v1363: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:38.720 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:38 smithi129 ceph-mon[30184]: pgmap v1363: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:38.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:38 smithi157 ceph-mon[33359]: pgmap v1363: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:40.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:40 smithi129 ceph-mon[30184]: pgmap v1364: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:40.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:40 smithi129 ceph-mon[42066]: pgmap v1364: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:40.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:40 smithi157 ceph-mon[33359]: pgmap v1364: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:42.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:42 smithi129 ceph-mon[30184]: pgmap v1365: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:42.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:42 smithi129 ceph-mon[42066]: pgmap v1365: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:42.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:42 smithi157 ceph-mon[33359]: pgmap v1365: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:44.528 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:14:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:14:44.529 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:44 smithi157 ceph-mon[33359]: pgmap v1366: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:44.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:44 smithi129 ceph-mon[30184]: pgmap v1366: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:44.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:44 smithi129 ceph-mon[42066]: pgmap v1366: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:45.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:45.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:45.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:46.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:46 smithi129 ceph-mon[30184]: pgmap v1367: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:46.469 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:46 smithi129 ceph-mon[42066]: pgmap v1367: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:46.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:14:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:14:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:14:46.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:46 smithi157 ceph-mon[33359]: pgmap v1367: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:48.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:48 smithi129 ceph-mon[30184]: pgmap v1368: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:48.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:48 smithi129 ceph-mon[42066]: pgmap v1368: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:48.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:48 smithi157 ceph-mon[33359]: pgmap v1368: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:50.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:50 smithi129 ceph-mon[42066]: pgmap v1369: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:50.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:50.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:50 smithi129 ceph-mon[30184]: pgmap v1369: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:50.720 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:50.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:50 smithi157 ceph-mon[33359]: pgmap v1369: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:50.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:14:52.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:52 smithi129 ceph-mon[42066]: pgmap v1370: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:52.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:52 smithi129 ceph-mon[30184]: pgmap v1370: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:52.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:52 smithi157 ceph-mon[33359]: pgmap v1370: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:14:53.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.254 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.255 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.256 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.257 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.257 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.257 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.257 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.257 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.257 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.257 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.257 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.257 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.257 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.258 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.258 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.258 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.258 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.258 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.258 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.258 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.259 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.260 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.261 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.262 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.263 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.264 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.264 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.264 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.264 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.264 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.265 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.265 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.265 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.265 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.265 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:53.265 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:14:54.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:54 smithi129 ceph-mon[42066]: pgmap v1371: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:54.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:54 smithi129 ceph-mon[30184]: pgmap v1371: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:54.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:14:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:14:54.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:54 smithi157 ceph-mon[33359]: pgmap v1371: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:55.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:55.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:55.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:14:56.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:56 smithi129 ceph-mon[30184]: pgmap v1372: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:56.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:14:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:14:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:14:56.471 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:56 smithi129 ceph-mon[42066]: pgmap v1372: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:56.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:56 smithi157 ceph-mon[33359]: pgmap v1372: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:14:58.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:14:58 smithi129 ceph-mon[42066]: pgmap v1373: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:58.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:14:58 smithi129 ceph-mon[30184]: pgmap v1373: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:14:58.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:14:58 smithi157 ceph-mon[33359]: pgmap v1373: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:00.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:00 smithi129 ceph-mon[30184]: pgmap v1374: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:00.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:00 smithi129 ceph-mon[42066]: pgmap v1374: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:00.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:00 smithi157 ceph-mon[33359]: pgmap v1374: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:02.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:02 smithi129 ceph-mon[30184]: pgmap v1375: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:02.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:02 smithi129 ceph-mon[42066]: pgmap v1375: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:02.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:02 smithi157 ceph-mon[33359]: pgmap v1375: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:04.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:04 smithi129 ceph-mon[30184]: pgmap v1376: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:04.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:04 smithi129 ceph-mon[42066]: pgmap v1376: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:04.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:15:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:15:04.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:04 smithi157 ceph-mon[33359]: pgmap v1376: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:05.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:05.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:05.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:05.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:05.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:05.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:05.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:05.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:05.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:15:06.469 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:15:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:15:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:15:06.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:06 smithi129 ceph-mon[30184]: pgmap v1377: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:06.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:06 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:06.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:06 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:06.470 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:06 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:15:06.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:06 smithi129 ceph-mon[42066]: pgmap v1377: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:06.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:06 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:06.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:06 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:06.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:06 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:15:06.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:06 smithi157 ceph-mon[33359]: pgmap v1377: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:06.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:06 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:15:06.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:06 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:15:06.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:06 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:15:08.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:08 smithi129 ceph-mon[30184]: pgmap v1378: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:08.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:08 smithi129 ceph-mon[42066]: pgmap v1378: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:08.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:08 smithi157 ceph-mon[33359]: pgmap v1378: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:10.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:10 smithi129 ceph-mon[30184]: pgmap v1379: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:10.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:10 smithi129 ceph-mon[42066]: pgmap v1379: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:10.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:10 smithi157 ceph-mon[33359]: pgmap v1379: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:12 smithi129 ceph-mon[30184]: pgmap v1380: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:12.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:12 smithi129 ceph-mon[42066]: pgmap v1380: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:12.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:12 smithi157 ceph-mon[33359]: pgmap v1380: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:14.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:14 smithi129 ceph-mon[30184]: pgmap v1381: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:14.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:14 smithi129 ceph-mon[42066]: pgmap v1381: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:14.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:15:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:15:14.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:14 smithi157 ceph-mon[33359]: pgmap v1381: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:15.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:15.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:15.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:16.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:16 smithi129 ceph-mon[30184]: pgmap v1382: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:16.469 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:16 smithi129 ceph-mon[42066]: pgmap v1382: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:16.470 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:15:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:15:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:15:16.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:16 smithi157 ceph-mon[33359]: pgmap v1382: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:18.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:18 smithi129 ceph-mon[30184]: pgmap v1383: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:18.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:18 smithi129 ceph-mon[42066]: pgmap v1383: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:18.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:18 smithi157 ceph-mon[33359]: pgmap v1383: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:20.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:20 smithi129 ceph-mon[30184]: pgmap v1384: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:20.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:20.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:20 smithi129 ceph-mon[42066]: pgmap v1384: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:20.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:20.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:20 smithi157 ceph-mon[33359]: pgmap v1384: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:20.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:22.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:22 smithi129 ceph-mon[30184]: pgmap v1385: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:22.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:22 smithi129 ceph-mon[42066]: pgmap v1385: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:22.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:22 smithi157 ceph-mon[33359]: pgmap v1385: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:24.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:24 smithi129 ceph-mon[30184]: pgmap v1386: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:24.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:24 smithi129 ceph-mon[42066]: pgmap v1386: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:24.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:24 smithi157 ceph-mon[33359]: pgmap v1386: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:24.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:15:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:15:25.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:25.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:25.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:26.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:26 smithi129 ceph-mon[30184]: pgmap v1387: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:26.469 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:15:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:15:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:15:26.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:26 smithi129 ceph-mon[42066]: pgmap v1387: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:26.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:26 smithi157 ceph-mon[33359]: pgmap v1387: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:28.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:28 smithi129 ceph-mon[30184]: pgmap v1388: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:28.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:28 smithi129 ceph-mon[42066]: pgmap v1388: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:28.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:28 smithi157 ceph-mon[33359]: pgmap v1388: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:30.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:30 smithi129 ceph-mon[30184]: pgmap v1389: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:30.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:30 smithi129 ceph-mon[42066]: pgmap v1389: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:30.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:30 smithi157 ceph-mon[33359]: pgmap v1389: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:32.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:32 smithi129 ceph-mon[30184]: pgmap v1390: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:32.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:32 smithi129 ceph-mon[42066]: pgmap v1390: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:32.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:32 smithi157 ceph-mon[33359]: pgmap v1390: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:34.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:34 smithi129 ceph-mon[30184]: pgmap v1391: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:34.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:34 smithi129 ceph-mon[42066]: pgmap v1391: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:34.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:34 smithi157 ceph-mon[33359]: pgmap v1391: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:34.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:15:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:15:35.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:35.764 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:35 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:35.764 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:35.765 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:35 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:35.765 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:35.765 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:35 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:36.350 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:15:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:15:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:15:36.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:36 smithi129 ceph-mon[30184]: pgmap v1392: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:36.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:36 smithi129 ceph-mon[42066]: pgmap v1392: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:36.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:36 smithi157 ceph-mon[33359]: pgmap v1392: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:38.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:38 smithi129 ceph-mon[30184]: pgmap v1393: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:38.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:38 smithi129 ceph-mon[42066]: pgmap v1393: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:38.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:38 smithi157 ceph-mon[33359]: pgmap v1393: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:40.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:40 smithi129 ceph-mon[30184]: pgmap v1394: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:40.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:40 smithi129 ceph-mon[42066]: pgmap v1394: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:40.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:40 smithi157 ceph-mon[33359]: pgmap v1394: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:42.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:42 smithi129 ceph-mon[30184]: pgmap v1395: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:42.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:42 smithi129 ceph-mon[42066]: pgmap v1395: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:42.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:42 smithi157 ceph-mon[33359]: pgmap v1395: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:44.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:44 smithi129 ceph-mon[30184]: pgmap v1396: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:44.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:44 smithi129 ceph-mon[42066]: pgmap v1396: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:44.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:44 smithi157 ceph-mon[33359]: pgmap v1396: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:44.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:15:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:15:45.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:45.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:45.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:46.371 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:15:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:15:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:15:46.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:46 smithi129 ceph-mon[30184]: pgmap v1397: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:46.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:46 smithi129 ceph-mon[42066]: pgmap v1397: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:46.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:46 smithi157 ceph-mon[33359]: pgmap v1397: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:15:48.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:48 smithi129 ceph-mon[30184]: pgmap v1398: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:48.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:48 smithi129 ceph-mon[42066]: pgmap v1398: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:48.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:48 smithi157 ceph-mon[33359]: pgmap v1398: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:50.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:50 smithi129 ceph-mon[30184]: pgmap v1399: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:50.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:50.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:50 smithi129 ceph-mon[42066]: pgmap v1399: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:50.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:50.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:50 smithi157 ceph-mon[33359]: pgmap v1399: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:50.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:15:52.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:52 smithi129 ceph-mon[30184]: pgmap v1400: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:52.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:52 smithi129 ceph-mon[42066]: pgmap v1400: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:52.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:52 smithi157 ceph-mon[33359]: pgmap v1400: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:54.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:54 smithi129 ceph-mon[30184]: pgmap v1401: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:54.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:54 smithi129 ceph-mon[42066]: pgmap v1401: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:54.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:54 smithi157 ceph-mon[33359]: pgmap v1401: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:54.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:15:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:15:55.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:55.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:55.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:15:56.392 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:15:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:15:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:15:56.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:56 smithi129 ceph-mon[30184]: pgmap v1402: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:56.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:56 smithi129 ceph-mon[42066]: pgmap v1402: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:56.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:56 smithi157 ceph-mon[33359]: pgmap v1402: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:15:58.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:15:58 smithi129 ceph-mon[42066]: pgmap v1403: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:58.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:15:58 smithi129 ceph-mon[30184]: pgmap v1403: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:15:58.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:15:58 smithi157 ceph-mon[33359]: pgmap v1403: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:00.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:00 smithi129 ceph-mon[30184]: pgmap v1404: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:00.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:00 smithi129 ceph-mon[42066]: pgmap v1404: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:00.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:00 smithi157 ceph-mon[33359]: pgmap v1404: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:02.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:02 smithi129 ceph-mon[30184]: pgmap v1405: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:02.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:02 smithi129 ceph-mon[42066]: pgmap v1405: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:02.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:02 smithi157 ceph-mon[33359]: pgmap v1405: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:04.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:04 smithi129 ceph-mon[30184]: pgmap v1406: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:04.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:04 smithi129 ceph-mon[42066]: pgmap v1406: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:04.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:16:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:16:04.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:04 smithi157 ceph-mon[33359]: pgmap v1406: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:05.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:05.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:05.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:05.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:05.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:05.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:06.410 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:16:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:16:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:16:06.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:06 smithi129 ceph-mon[30184]: pgmap v1407: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:06.873 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:06 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:06.874 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:06 smithi129 ceph-mon[42066]: pgmap v1407: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:06.874 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:06 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:06.875 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:06 smithi157 ceph-mon[33359]: pgmap v1407: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:06.875 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:06 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:16:07.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:07 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:07.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:07 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:07.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:07 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:16:07.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:07 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:07.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:07 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:07.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:07 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:16:07.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:07 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:16:07.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:07 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:16:07.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:07 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:16:08.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:08 smithi129 ceph-mon[30184]: pgmap v1408: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:08.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:08 smithi129 ceph-mon[42066]: pgmap v1408: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:08.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:08 smithi157 ceph-mon[33359]: pgmap v1408: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:10.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:10 smithi129 ceph-mon[30184]: pgmap v1409: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:10.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:10 smithi129 ceph-mon[42066]: pgmap v1409: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:10.976 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:10 smithi157 ceph-mon[33359]: pgmap v1409: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:12.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:12 smithi129 ceph-mon[30184]: pgmap v1410: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:12.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:12 smithi129 ceph-mon[42066]: pgmap v1410: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:12.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:12 smithi157 ceph-mon[33359]: pgmap v1410: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:14.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:16:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:16:14.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:14 smithi157 ceph-mon[33359]: pgmap v1411: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:14.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:14 smithi129 ceph-mon[42066]: pgmap v1411: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:14.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:14 smithi129 ceph-mon[30184]: pgmap v1411: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:15.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:15.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:15.976 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:16.468 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:16:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:16:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:16:16.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:16 smithi129 ceph-mon[30184]: pgmap v1412: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:16.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:16 smithi129 ceph-mon[42066]: pgmap v1412: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:16.976 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:16 smithi157 ceph-mon[33359]: pgmap v1412: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:18.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:18 smithi129 ceph-mon[30184]: pgmap v1413: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:18.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:18 smithi129 ceph-mon[42066]: pgmap v1413: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:18.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:18 smithi157 ceph-mon[33359]: pgmap v1413: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:19.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:19.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:19.976 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:20.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:20 smithi129 ceph-mon[30184]: pgmap v1414: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:20.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:20 smithi129 ceph-mon[42066]: pgmap v1414: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:20.976 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:20 smithi157 ceph-mon[33359]: pgmap v1414: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:22.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:22 smithi129 ceph-mon[30184]: pgmap v1415: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:22.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:22 smithi129 ceph-mon[42066]: pgmap v1415: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:22.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:22 smithi157 ceph-mon[33359]: pgmap v1415: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:24.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:16:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:16:24.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:24 smithi157 ceph-mon[33359]: pgmap v1416: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:24.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:24 smithi129 ceph-mon[30184]: pgmap v1416: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:24.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:24 smithi129 ceph-mon[42066]: pgmap v1416: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:25.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:25.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:25.976 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:26.468 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:16:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:16:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:16:26.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:26 smithi129 ceph-mon[30184]: pgmap v1417: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:26.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:26 smithi129 ceph-mon[42066]: pgmap v1417: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:26.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:26 smithi157 ceph-mon[33359]: pgmap v1417: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:28.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:28 smithi129 ceph-mon[30184]: pgmap v1418: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:28.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:28 smithi129 ceph-mon[42066]: pgmap v1418: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:28.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:28 smithi157 ceph-mon[33359]: pgmap v1418: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:30.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:30 smithi129 ceph-mon[30184]: pgmap v1419: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:30.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:30 smithi129 ceph-mon[42066]: pgmap v1419: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:30.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:30 smithi157 ceph-mon[33359]: pgmap v1419: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:32.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:32 smithi129 ceph-mon[42066]: pgmap v1420: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:32.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:32 smithi129 ceph-mon[30184]: pgmap v1420: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:32.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:32 smithi157 ceph-mon[33359]: pgmap v1420: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:33.265 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.265 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.265 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.266 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.266 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.266 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.266 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.266 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.266 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.266 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.266 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.266 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.266 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.266 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.267 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.267 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.267 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.267 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.267 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.267 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.267 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.267 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.267 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.267 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.267 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.267 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.267 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.267 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.268 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.268 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.268 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.268 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.268 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.268 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.268 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.268 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.268 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.268 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.268 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.268 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.268 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.268 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.269 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.270 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.271 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.272 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:33.273 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:16:34.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:16:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:16:34.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:34 smithi157 ceph-mon[33359]: pgmap v1421: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:34.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:34 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:34.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:34 smithi129 ceph-mon[30184]: pgmap v1421: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:34.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:34 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:34.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:34 smithi129 ceph-mon[42066]: pgmap v1421: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:34.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:34 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:35.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:35.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:35.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:36.468 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:16:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:16:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:16:36.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:36 smithi129 ceph-mon[42066]: pgmap v1422: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:36.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:36 smithi129 ceph-mon[30184]: pgmap v1422: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:36.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:36 smithi157 ceph-mon[33359]: pgmap v1422: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:38.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:38 smithi129 ceph-mon[30184]: pgmap v1423: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:38.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:38 smithi129 ceph-mon[42066]: pgmap v1423: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:38.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:38 smithi157 ceph-mon[33359]: pgmap v1423: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:40.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:40 smithi129 ceph-mon[30184]: pgmap v1424: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:40.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:40 smithi129 ceph-mon[42066]: pgmap v1424: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:40.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:40 smithi157 ceph-mon[33359]: pgmap v1424: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:42.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:42 smithi129 ceph-mon[30184]: pgmap v1425: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:42.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:42 smithi129 ceph-mon[42066]: pgmap v1425: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:42.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:42 smithi157 ceph-mon[33359]: pgmap v1425: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:44.725 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:16:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:16:44.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:44 smithi157 ceph-mon[33359]: pgmap v1426: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:44.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:44 smithi129 ceph-mon[42066]: pgmap v1426: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:44.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:44 smithi129 ceph-mon[30184]: pgmap v1426: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:45.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:45.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:45.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:46.468 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:16:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:16:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:16:46.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:46 smithi129 ceph-mon[30184]: pgmap v1427: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:46.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:46 smithi129 ceph-mon[42066]: pgmap v1427: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:46.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:46 smithi157 ceph-mon[33359]: pgmap v1427: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:48.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:48 smithi129 ceph-mon[30184]: pgmap v1428: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:48.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:48 smithi129 ceph-mon[42066]: pgmap v1428: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:48.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:48 smithi157 ceph-mon[33359]: pgmap v1428: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:49.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:49 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:49.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:49 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:49.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:49 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:16:50.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:50 smithi129 ceph-mon[42066]: pgmap v1429: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:50.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:50 smithi129 ceph-mon[30184]: pgmap v1429: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:50.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:50 smithi157 ceph-mon[33359]: pgmap v1429: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:52.969 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:52 smithi129 ceph-mon[30184]: pgmap v1430: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:52.970 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:52 smithi129 ceph-mon[42066]: pgmap v1430: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:52.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:52 smithi157 ceph-mon[33359]: pgmap v1430: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:16:54.725 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:16:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:16:54.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:54 smithi157 ceph-mon[33359]: pgmap v1431: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:54.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:54 smithi129 ceph-mon[30184]: pgmap v1431: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:54.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:54 smithi129 ceph-mon[42066]: pgmap v1431: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:55.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:55.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:55.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:16:56.468 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:16:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:16:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:16:56.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:56 smithi129 ceph-mon[30184]: pgmap v1432: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:56.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:56 smithi129 ceph-mon[42066]: pgmap v1432: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:56.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:56 smithi157 ceph-mon[33359]: pgmap v1432: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:16:58.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:16:58 smithi129 ceph-mon[30184]: pgmap v1433: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:58.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:16:58 smithi129 ceph-mon[42066]: pgmap v1433: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:16:58.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:16:58 smithi157 ceph-mon[33359]: pgmap v1433: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:00.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:00 smithi129 ceph-mon[30184]: pgmap v1434: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:00.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:00 smithi129 ceph-mon[42066]: pgmap v1434: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:00.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:00 smithi157 ceph-mon[33359]: pgmap v1434: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:02.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:02 smithi129 ceph-mon[30184]: pgmap v1435: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:02.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:02 smithi129 ceph-mon[42066]: pgmap v1435: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:02.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:02 smithi157 ceph-mon[33359]: pgmap v1435: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:04.725 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:17:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:17:04.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:04 smithi157 ceph-mon[33359]: pgmap v1436: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:04.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:04.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:04 smithi129 ceph-mon[30184]: pgmap v1436: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:04.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:04.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:04 smithi129 ceph-mon[42066]: pgmap v1436: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:04.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:05.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:05.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:05.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:06.468 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:17:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:17:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:17:06.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:06 smithi129 ceph-mon[30184]: pgmap v1437: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:06.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:06 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:06.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:06 smithi129 ceph-mon[42066]: pgmap v1437: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:06.969 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:06 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:06.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:06 smithi157 ceph-mon[33359]: pgmap v1437: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:06.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:06 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:17:08.468 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:08 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:17:08.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:08 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:17:08.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:08 smithi129 ceph-mon[30184]: pgmap v1438: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:08.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:08 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:08.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:08 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:08.469 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:08 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:17:08.469 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:08 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:17:08.469 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:08 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:17:08.469 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:08 smithi129 ceph-mon[42066]: pgmap v1438: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:08.469 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:08 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:08.469 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:08 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:08.470 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:08 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:17:08.475 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:08 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:17:08.475 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:08 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:17:08.475 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:08 smithi157 ceph-mon[33359]: pgmap v1438: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:08.475 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:08 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:17:08.475 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:08 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:17:08.476 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:08 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:17:10.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:10 smithi129 ceph-mon[30184]: pgmap v1439: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:10.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:10 smithi129 ceph-mon[42066]: pgmap v1439: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:10.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:10 smithi157 ceph-mon[33359]: pgmap v1439: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:12.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:12 smithi129 ceph-mon[30184]: pgmap v1440: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:12.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:12 smithi129 ceph-mon[42066]: pgmap v1440: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:12.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:12 smithi157 ceph-mon[33359]: pgmap v1440: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:14.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:14 smithi129 ceph-mon[30184]: pgmap v1441: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:14.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:14 smithi129 ceph-mon[42066]: pgmap v1441: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:14.725 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:17:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:17:14.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:14 smithi157 ceph-mon[33359]: pgmap v1441: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:15.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:15.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:15.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:16.468 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:16 smithi129 ceph-mon[30184]: pgmap v1442: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:16.469 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:16 smithi129 ceph-mon[42066]: pgmap v1442: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:16.469 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:17:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:17:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:17:16.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:16 smithi157 ceph-mon[33359]: pgmap v1442: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:18.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:18 smithi129 ceph-mon[30184]: pgmap v1443: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:18.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:18 smithi129 ceph-mon[42066]: pgmap v1443: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:18.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:18 smithi157 ceph-mon[33359]: pgmap v1443: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:20.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:20 smithi129 ceph-mon[30184]: pgmap v1444: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:20.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:20.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:20 smithi129 ceph-mon[42066]: pgmap v1444: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:20.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:20.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:20 smithi157 ceph-mon[33359]: pgmap v1444: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:20.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:22.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:22 smithi129 ceph-mon[30184]: pgmap v1445: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:22.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:22 smithi129 ceph-mon[42066]: pgmap v1445: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:22.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:22 smithi157 ceph-mon[33359]: pgmap v1445: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:24.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:24 smithi129 ceph-mon[30184]: pgmap v1446: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:24.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:24 smithi129 ceph-mon[42066]: pgmap v1446: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:24.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:17:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:17:24.726 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:24 smithi157 ceph-mon[33359]: pgmap v1446: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:25.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:25.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:25.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:26.468 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:26 smithi129 ceph-mon[30184]: pgmap v1447: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:26.469 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:26 smithi129 ceph-mon[42066]: pgmap v1447: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:26.469 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:17:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:17:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:17:26.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:26 smithi157 ceph-mon[33359]: pgmap v1447: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:28.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:28 smithi129 ceph-mon[42066]: pgmap v1448: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:28.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:28 smithi129 ceph-mon[30184]: pgmap v1448: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:28.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:28 smithi157 ceph-mon[33359]: pgmap v1448: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:30.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:30 smithi129 ceph-mon[30184]: pgmap v1449: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:30.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:30 smithi129 ceph-mon[42066]: pgmap v1449: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:30.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:30 smithi157 ceph-mon[33359]: pgmap v1449: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:32.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:32 smithi129 ceph-mon[30184]: pgmap v1450: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:32.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:32 smithi129 ceph-mon[42066]: pgmap v1450: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:32.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:32 smithi157 ceph-mon[33359]: pgmap v1450: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:34.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:34 smithi129 ceph-mon[30184]: pgmap v1451: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:34.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:34 smithi129 ceph-mon[42066]: pgmap v1451: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:34.725 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:17:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:17:34.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:34 smithi157 ceph-mon[33359]: pgmap v1451: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:35.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:35.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:35 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:35.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:35.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:35 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:35.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:35.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:35 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:36.468 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:36 smithi129 ceph-mon[30184]: pgmap v1452: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:36.468 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:17:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:17:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:17:36.469 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:36 smithi129 ceph-mon[42066]: pgmap v1452: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:36.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:36 smithi157 ceph-mon[33359]: pgmap v1452: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:38.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:38 smithi129 ceph-mon[30184]: pgmap v1453: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:38.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:38 smithi129 ceph-mon[42066]: pgmap v1453: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:38.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:38 smithi157 ceph-mon[33359]: pgmap v1453: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:40.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:40 smithi129 ceph-mon[30184]: pgmap v1454: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:40.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:40 smithi129 ceph-mon[42066]: pgmap v1454: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:40.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:40 smithi157 ceph-mon[33359]: pgmap v1454: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:42.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:42 smithi129 ceph-mon[30184]: pgmap v1455: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:42.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:42 smithi129 ceph-mon[42066]: pgmap v1455: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:42.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:42 smithi157 ceph-mon[33359]: pgmap v1455: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:44.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:44 smithi129 ceph-mon[30184]: pgmap v1456: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:44.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:44 smithi129 ceph-mon[42066]: pgmap v1456: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:44.725 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:17:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:17:44.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:44 smithi157 ceph-mon[33359]: pgmap v1456: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:45.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:45.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:45.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:46.351 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:17:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:17:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:17:46.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:46 smithi129 ceph-mon[30184]: pgmap v1457: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:46.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:46 smithi129 ceph-mon[42066]: pgmap v1457: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:46.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:46 smithi157 ceph-mon[33359]: pgmap v1457: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:48.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:48 smithi129 ceph-mon[30184]: pgmap v1458: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:48.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:48 smithi129 ceph-mon[42066]: pgmap v1458: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:48.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:48 smithi157 ceph-mon[33359]: pgmap v1458: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:50.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:50 smithi129 ceph-mon[30184]: pgmap v1459: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:50.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:50.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:50 smithi129 ceph-mon[42066]: pgmap v1459: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:50.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:50.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:50 smithi157 ceph-mon[33359]: pgmap v1459: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:50.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:17:52.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:52 smithi129 ceph-mon[30184]: pgmap v1460: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:52.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:52 smithi129 ceph-mon[42066]: pgmap v1460: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:52.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:52 smithi157 ceph-mon[33359]: pgmap v1460: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:17:54.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:54 smithi129 ceph-mon[30184]: pgmap v1461: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:54.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:54 smithi129 ceph-mon[42066]: pgmap v1461: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:54.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:54 smithi157 ceph-mon[33359]: pgmap v1461: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:54.725 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:17:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:17:55.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:55.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:55.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:17:56.372 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:17:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:17:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:17:56.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:56 smithi129 ceph-mon[30184]: pgmap v1462: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:56.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:56 smithi129 ceph-mon[42066]: pgmap v1462: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:56.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:56 smithi157 ceph-mon[33359]: pgmap v1462: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:17:58.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:17:58 smithi129 ceph-mon[30184]: pgmap v1463: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:58.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:17:58 smithi129 ceph-mon[42066]: pgmap v1463: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:17:58.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:17:58 smithi157 ceph-mon[33359]: pgmap v1463: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:00.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:00 smithi129 ceph-mon[30184]: pgmap v1464: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:00.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:00 smithi129 ceph-mon[42066]: pgmap v1464: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:00.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:00 smithi157 ceph-mon[33359]: pgmap v1464: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:02.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:02 smithi129 ceph-mon[42066]: pgmap v1465: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:02.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:02 smithi129 ceph-mon[30184]: pgmap v1465: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:02.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:02 smithi157 ceph-mon[33359]: pgmap v1465: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:04.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:04 smithi129 ceph-mon[30184]: pgmap v1466: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:04.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:04 smithi129 ceph-mon[42066]: pgmap v1466: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:04.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:04 smithi157 ceph-mon[33359]: pgmap v1466: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:04.725 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:18:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:18:05.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:05.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:05.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:05.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:05.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:05.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:06.394 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:18:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:18:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:18:06.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:06 smithi129 ceph-mon[30184]: pgmap v1467: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:06.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:06 smithi129 ceph-mon[42066]: pgmap v1467: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:06.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:06 smithi157 ceph-mon[33359]: pgmap v1467: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:08.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:08 smithi129 ceph-mon[30184]: pgmap v1468: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:08.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:08 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:08.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:08 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:18:08.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:08 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:18:08.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:08 smithi129 ceph-mon[42066]: pgmap v1468: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:08.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:08 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:08.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:08 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:18:08.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:08 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:18:08.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:08 smithi157 ceph-mon[33359]: pgmap v1468: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:08.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:08 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:18:08.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:08 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:18:08.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:08 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:18:09.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:09 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:09.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:09 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:09.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:09 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:18:09.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:09 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:09.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:09 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:09.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:09 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:18:09.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:09 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:18:09.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:09 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:18:09.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:09 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:18:10.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:10 smithi129 ceph-mon[30184]: pgmap v1469: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:10.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:10 smithi129 ceph-mon[42066]: pgmap v1469: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:10.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:10 smithi157 ceph-mon[33359]: pgmap v1469: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:12.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:12 smithi129 ceph-mon[30184]: pgmap v1470: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:12.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:12 smithi129 ceph-mon[42066]: pgmap v1470: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:12.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:12 smithi157 ceph-mon[33359]: pgmap v1470: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:13.277 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.277 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.277 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.277 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.278 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.279 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.280 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.281 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.281 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.281 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.281 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.281 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.281 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.281 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.281 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.281 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.281 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.281 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.281 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.281 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.281 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.282 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.282 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.282 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.282 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.282 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.282 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.282 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.282 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.282 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.282 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.282 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.282 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.282 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.283 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.283 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.283 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.283 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.283 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.283 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.283 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.283 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.283 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.283 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.283 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.283 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.283 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.284 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.284 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.284 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.284 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.284 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.284 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.284 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.284 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.284 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.285 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.285 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.285 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.285 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.285 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.285 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.285 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.285 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.285 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.285 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:13.285 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:18:14.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:14 smithi129 ceph-mon[30184]: pgmap v1471: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:14.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:14 smithi129 ceph-mon[42066]: pgmap v1471: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:14.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:14 smithi157 ceph-mon[33359]: pgmap v1471: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:14.725 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:18:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:18:15.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:15.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:15.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:16.412 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:18:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:18:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:18:16.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:16 smithi129 ceph-mon[30184]: pgmap v1472: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:16.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:16 smithi129 ceph-mon[42066]: pgmap v1472: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:16.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:16 smithi157 ceph-mon[33359]: pgmap v1472: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:18.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:18 smithi129 ceph-mon[30184]: pgmap v1473: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:18.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:18 smithi129 ceph-mon[42066]: pgmap v1473: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:18.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:18 smithi157 ceph-mon[33359]: pgmap v1473: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:20.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:20 smithi129 ceph-mon[30184]: pgmap v1474: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:20.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:20.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:20 smithi129 ceph-mon[42066]: pgmap v1474: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:20.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:20.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:20 smithi157 ceph-mon[33359]: pgmap v1474: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:20.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:22.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:22 smithi129 ceph-mon[30184]: pgmap v1475: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:22.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:22 smithi129 ceph-mon[42066]: pgmap v1475: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:22.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:22 smithi157 ceph-mon[33359]: pgmap v1475: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:24.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:24 smithi129 ceph-mon[30184]: pgmap v1476: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:24.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:24 smithi129 ceph-mon[42066]: pgmap v1476: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:24.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:24 smithi157 ceph-mon[33359]: pgmap v1476: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:24.725 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:18:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:18:25.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:25.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:25.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:26.433 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:18:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:18:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:18:26.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:26 smithi129 ceph-mon[30184]: pgmap v1477: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:26.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:26 smithi129 ceph-mon[42066]: pgmap v1477: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:26.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:26 smithi157 ceph-mon[33359]: pgmap v1477: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:28.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:28 smithi129 ceph-mon[30184]: pgmap v1478: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:28.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:28 smithi129 ceph-mon[42066]: pgmap v1478: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:28.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:28 smithi157 ceph-mon[33359]: pgmap v1478: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:30.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:30 smithi129 ceph-mon[30184]: pgmap v1479: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:30.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:30 smithi129 ceph-mon[42066]: pgmap v1479: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:30.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:30 smithi157 ceph-mon[33359]: pgmap v1479: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:32.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:32 smithi129 ceph-mon[30184]: pgmap v1480: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:32.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:32 smithi129 ceph-mon[42066]: pgmap v1480: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:32.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:32 smithi157 ceph-mon[33359]: pgmap v1480: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:34.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:34 smithi129 ceph-mon[30184]: pgmap v1481: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:34.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:34 smithi129 ceph-mon[42066]: pgmap v1481: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:34.725 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:18:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:18:34.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:34 smithi157 ceph-mon[33359]: pgmap v1481: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:35.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:35.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:35 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:35.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:35.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:35 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:35.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:35.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:35 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:36.454 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:18:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:18:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:18:36.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:36 smithi129 ceph-mon[30184]: pgmap v1482: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:36.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:36 smithi129 ceph-mon[42066]: pgmap v1482: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:36.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:36 smithi157 ceph-mon[33359]: pgmap v1482: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:38.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:38 smithi129 ceph-mon[30184]: pgmap v1483: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:38.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:38 smithi129 ceph-mon[42066]: pgmap v1483: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:38.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:38 smithi157 ceph-mon[33359]: pgmap v1483: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:40.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:40 smithi129 ceph-mon[30184]: pgmap v1484: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:40.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:40 smithi129 ceph-mon[42066]: pgmap v1484: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:40.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:40 smithi157 ceph-mon[33359]: pgmap v1484: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:42.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:42 smithi129 ceph-mon[30184]: pgmap v1485: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:42.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:42 smithi129 ceph-mon[42066]: pgmap v1485: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:42.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:42 smithi157 ceph-mon[33359]: pgmap v1485: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:44.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:44 smithi129 ceph-mon[30184]: pgmap v1486: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:44.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:44 smithi129 ceph-mon[42066]: pgmap v1486: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:44.725 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:18:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:18:44.725 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:44 smithi157 ceph-mon[33359]: pgmap v1486: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:45.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:45.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:45.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:46.467 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:18:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:18:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:18:46.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:46 smithi129 ceph-mon[30184]: pgmap v1487: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:46.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:46 smithi129 ceph-mon[42066]: pgmap v1487: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:46.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:46 smithi157 ceph-mon[33359]: pgmap v1487: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:48.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:48 smithi129 ceph-mon[30184]: pgmap v1488: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:48.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:48 smithi129 ceph-mon[42066]: pgmap v1488: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:48.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:48 smithi157 ceph-mon[33359]: pgmap v1488: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:50.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:50 smithi129 ceph-mon[30184]: pgmap v1489: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:50.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:50.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:50 smithi129 ceph-mon[42066]: pgmap v1489: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:50.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:50.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:50 smithi157 ceph-mon[33359]: pgmap v1489: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:50.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:18:52.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:52 smithi129 ceph-mon[30184]: pgmap v1490: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:52.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:52 smithi129 ceph-mon[42066]: pgmap v1490: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:52.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:52 smithi157 ceph-mon[33359]: pgmap v1490: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:18:54.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:54 smithi129 ceph-mon[30184]: pgmap v1491: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:54.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:54 smithi129 ceph-mon[42066]: pgmap v1491: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:54.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:18:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:18:54.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:54 smithi157 ceph-mon[33359]: pgmap v1491: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:55.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:55.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:55.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:18:56.468 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:18:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:18:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:18:56.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:56 smithi129 ceph-mon[42066]: pgmap v1492: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:56.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:56 smithi129 ceph-mon[30184]: pgmap v1492: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:56.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:56 smithi157 ceph-mon[33359]: pgmap v1492: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:18:58.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:18:58 smithi129 ceph-mon[30184]: pgmap v1493: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:58.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:18:58 smithi129 ceph-mon[42066]: pgmap v1493: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:18:58.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:18:58 smithi157 ceph-mon[33359]: pgmap v1493: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:00.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:00 smithi129 ceph-mon[30184]: pgmap v1494: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:00.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:00 smithi129 ceph-mon[42066]: pgmap v1494: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:00.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:00 smithi157 ceph-mon[33359]: pgmap v1494: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:02.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:02 smithi129 ceph-mon[30184]: pgmap v1495: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:02.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:02 smithi129 ceph-mon[42066]: pgmap v1495: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:02.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:02 smithi157 ceph-mon[33359]: pgmap v1495: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:04.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:04 smithi129 ceph-mon[30184]: pgmap v1496: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:04.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:04 smithi129 ceph-mon[42066]: pgmap v1496: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:04.975 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:19:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:19:04.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:04 smithi157 ceph-mon[33359]: pgmap v1496: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:05.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:05.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:05.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:05.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:05.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:05.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:06.467 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:19:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:19:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:19:06.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:06 smithi129 ceph-mon[30184]: pgmap v1497: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:06.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:06 smithi129 ceph-mon[42066]: pgmap v1497: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:06.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:06 smithi157 ceph-mon[33359]: pgmap v1497: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:08.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:08 smithi129 ceph-mon[30184]: pgmap v1498: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:08.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:08 smithi129 ceph-mon[42066]: pgmap v1498: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:08.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:08 smithi157 ceph-mon[33359]: pgmap v1498: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:09.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:09 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:09.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:09 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:09.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:09 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:19:10.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:10 smithi129 ceph-mon[30184]: pgmap v1499: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:10.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:10 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:10.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:10 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:19:10.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:10 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:19:10.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:10 smithi129 ceph-mon[42066]: pgmap v1499: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:10.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:10 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:10.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:10 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:19:10.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:10 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:19:10.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:10 smithi157 ceph-mon[33359]: pgmap v1499: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:10.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:10 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:19:10.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:10 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:19:10.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:10 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:19:12.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:12 smithi129 ceph-mon[30184]: pgmap v1500: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:12.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:12 smithi129 ceph-mon[42066]: pgmap v1500: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:12.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:12 smithi157 ceph-mon[33359]: pgmap v1500: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:14.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:14 smithi129 ceph-mon[30184]: pgmap v1501: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:14.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:14 smithi129 ceph-mon[42066]: pgmap v1501: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:14.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:14 smithi157 ceph-mon[33359]: pgmap v1501: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:14.975 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:19:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:19:15.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:15.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:15.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:16.467 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:19:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:19:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:19:16.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:16 smithi129 ceph-mon[30184]: pgmap v1502: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:16.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:16 smithi129 ceph-mon[42066]: pgmap v1502: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:16.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:16 smithi157 ceph-mon[33359]: pgmap v1502: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:18.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:18 smithi129 ceph-mon[30184]: pgmap v1503: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:18.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:18 smithi129 ceph-mon[42066]: pgmap v1503: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:18.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:18 smithi157 ceph-mon[33359]: pgmap v1503: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:20.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:20 smithi129 ceph-mon[42066]: pgmap v1504: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:20.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:20.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:20 smithi129 ceph-mon[30184]: pgmap v1504: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:20.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:20.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:20 smithi157 ceph-mon[33359]: pgmap v1504: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:20.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:22.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:22 smithi129 ceph-mon[30184]: pgmap v1505: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:22.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:22 smithi129 ceph-mon[42066]: pgmap v1505: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:22.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:22 smithi157 ceph-mon[33359]: pgmap v1505: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:24.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:24 smithi129 ceph-mon[30184]: pgmap v1506: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:24.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:24 smithi129 ceph-mon[42066]: pgmap v1506: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:24.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:19:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:19:24.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:24 smithi157 ceph-mon[33359]: pgmap v1506: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:25.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:25.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:25.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:26.467 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:19:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:19:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:19:26.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:26 smithi129 ceph-mon[30184]: pgmap v1507: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:26.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:26 smithi129 ceph-mon[42066]: pgmap v1507: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:26.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:26 smithi157 ceph-mon[33359]: pgmap v1507: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:28.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:28 smithi129 ceph-mon[30184]: pgmap v1508: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:28.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:28 smithi129 ceph-mon[42066]: pgmap v1508: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:28.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:28 smithi157 ceph-mon[33359]: pgmap v1508: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:30.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:30 smithi129 ceph-mon[30184]: pgmap v1509: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:30.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:30 smithi129 ceph-mon[42066]: pgmap v1509: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:30.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:30 smithi157 ceph-mon[33359]: pgmap v1509: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:32.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:32 smithi129 ceph-mon[30184]: pgmap v1510: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:32.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:32 smithi129 ceph-mon[42066]: pgmap v1510: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:32.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:32 smithi157 ceph-mon[33359]: pgmap v1510: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:34.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:34 smithi129 ceph-mon[30184]: pgmap v1511: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:34.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:34 smithi129 ceph-mon[42066]: pgmap v1511: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:34.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:34 smithi157 ceph-mon[33359]: pgmap v1511: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:34.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:19:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:19:35.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:35.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:35 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:35.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:35.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:35 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:35.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:35.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:35 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:36.467 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:19:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:19:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:19:36.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:36 smithi129 ceph-mon[30184]: pgmap v1512: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:36.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:36 smithi129 ceph-mon[42066]: pgmap v1512: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:36.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:36 smithi157 ceph-mon[33359]: pgmap v1512: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:38.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:38 smithi129 ceph-mon[30184]: pgmap v1513: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:38.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:38 smithi129 ceph-mon[42066]: pgmap v1513: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:38.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:38 smithi157 ceph-mon[33359]: pgmap v1513: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:40.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:40 smithi129 ceph-mon[30184]: pgmap v1514: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:40.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:40 smithi129 ceph-mon[42066]: pgmap v1514: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:40.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:40 smithi157 ceph-mon[33359]: pgmap v1514: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:42.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:42 smithi129 ceph-mon[30184]: pgmap v1515: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:42.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:42 smithi129 ceph-mon[42066]: pgmap v1515: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:42.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:42 smithi157 ceph-mon[33359]: pgmap v1515: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:44.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:44 smithi129 ceph-mon[30184]: pgmap v1516: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:44.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:44 smithi129 ceph-mon[42066]: pgmap v1516: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:44.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:44 smithi157 ceph-mon[33359]: pgmap v1516: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:44.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:19:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:19:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:45.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:45.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:46.467 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:19:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:19:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:19:46.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:46 smithi129 ceph-mon[42066]: pgmap v1517: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:46.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:46 smithi129 ceph-mon[30184]: pgmap v1517: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:46.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:46 smithi157 ceph-mon[33359]: pgmap v1517: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:48.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:48 smithi129 ceph-mon[30184]: pgmap v1518: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:48.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:48 smithi129 ceph-mon[42066]: pgmap v1518: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:48.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:48 smithi157 ceph-mon[33359]: pgmap v1518: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:50.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:50 smithi129 ceph-mon[30184]: pgmap v1519: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:50.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:50.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:50 smithi129 ceph-mon[42066]: pgmap v1519: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:50.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:50.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:50 smithi157 ceph-mon[33359]: pgmap v1519: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:50.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:19:52.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:52 smithi129 ceph-mon[30184]: pgmap v1520: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:52.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:52 smithi129 ceph-mon[42066]: pgmap v1520: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:52.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:52 smithi157 ceph-mon[33359]: pgmap v1520: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:19:53.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.288 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.289 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.290 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.291 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.292 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.293 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.294 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.295 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.296 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.297 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.298 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:53.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:19:54.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:54 smithi129 ceph-mon[30184]: pgmap v1521: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:54.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:54 smithi129 ceph-mon[42066]: pgmap v1521: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:54.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:19:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:19:54.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:54 smithi157 ceph-mon[33359]: pgmap v1521: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:55.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:55.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:55.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:19:56.467 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:19:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:19:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:19:56.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:56 smithi129 ceph-mon[30184]: pgmap v1522: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:56.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:56 smithi129 ceph-mon[42066]: pgmap v1522: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:56.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:56 smithi157 ceph-mon[33359]: pgmap v1522: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:19:58.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:19:58 smithi129 ceph-mon[30184]: pgmap v1523: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:58.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:19:58 smithi129 ceph-mon[42066]: pgmap v1523: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:19:58.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:19:58 smithi157 ceph-mon[33359]: pgmap v1523: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:00.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:00 smithi129 ceph-mon[30184]: pgmap v1524: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:00.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:00 smithi129 ceph-mon[30184]: overall HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-11T21:20:00.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:00 smithi129 ceph-mon[42066]: pgmap v1524: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:00.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:00 smithi129 ceph-mon[42066]: overall HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-11T21:20:00.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:00 smithi157 ceph-mon[33359]: pgmap v1524: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:00.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:00 smithi157 ceph-mon[33359]: overall HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-11T21:20:02.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:02 smithi129 ceph-mon[30184]: pgmap v1525: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:02.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:02 smithi129 ceph-mon[42066]: pgmap v1525: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:02.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:02 smithi157 ceph-mon[33359]: pgmap v1525: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:04.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:04 smithi129 ceph-mon[30184]: pgmap v1526: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:04.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:04 smithi129 ceph-mon[42066]: pgmap v1526: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:04.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:04 smithi157 ceph-mon[33359]: pgmap v1526: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:04.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:20:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:20:05.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:05.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:05.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:05.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:05.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:05.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:06.467 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:20:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:20:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:20:06.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:06 smithi129 ceph-mon[42066]: pgmap v1527: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:06.968 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:06 smithi129 ceph-mon[30184]: pgmap v1527: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:06.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:06 smithi157 ceph-mon[33359]: pgmap v1527: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:08.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:08 smithi129 ceph-mon[30184]: pgmap v1528: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:08.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:08 smithi129 ceph-mon[42066]: pgmap v1528: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:08.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:08 smithi157 ceph-mon[33359]: pgmap v1528: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:10.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:10 smithi129 ceph-mon[30184]: pgmap v1529: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:10.914 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:10 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:10.915 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:10 smithi129 ceph-mon[42066]: pgmap v1529: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:10.915 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:10 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:10.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:10 smithi157 ceph-mon[33359]: pgmap v1529: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:10.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:10 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:12.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:12 smithi129 ceph-mon[30184]: pgmap v1530: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:12.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:12 smithi129 ceph-mon[42066]: pgmap v1530: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:12.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:12 smithi157 ceph-mon[33359]: pgmap v1530: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:14.939 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:20:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:20:15.217 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:14 smithi129 ceph-mon[30184]: pgmap v1531: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:15.217 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:14 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:20:15.217 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:14 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:20:15.218 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:14 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:15.218 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:14 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:20:15.218 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:14 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:20:15.218 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:14 smithi129 ceph-mon[42066]: pgmap v1531: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:15.218 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:14 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:20:15.218 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:14 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:20:15.218 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:14 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:15.218 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:14 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:20:15.218 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:14 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:20:15.224 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:14 smithi157 ceph-mon[33359]: pgmap v1531: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:15.225 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:14 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:20:15.225 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:14 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:20:15.225 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:14 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:15.225 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:14 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:20:15.225 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:14 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:20:16.217 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:16.218 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:20:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:20:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:20:16.218 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:16.224 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:17.217 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:16 smithi129 ceph-mon[30184]: pgmap v1532: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:17.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:16 smithi129 ceph-mon[42066]: pgmap v1532: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:17.224 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:16 smithi157 ceph-mon[33359]: pgmap v1532: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:18.467 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:18 smithi129 ceph-mon[30184]: pgmap v1533: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:18.467 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:18 smithi129 ceph-mon[42066]: pgmap v1533: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:18.474 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:18 smithi157 ceph-mon[33359]: pgmap v1533: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:20.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:20 smithi129 ceph-mon[30184]: pgmap v1534: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:20.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:20.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:20 smithi129 ceph-mon[42066]: pgmap v1534: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:20.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:20.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:20 smithi157 ceph-mon[33359]: pgmap v1534: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:20.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:22.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:22 smithi129 ceph-mon[30184]: pgmap v1535: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:22.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:22 smithi129 ceph-mon[42066]: pgmap v1535: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:22.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:22 smithi157 ceph-mon[33359]: pgmap v1535: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:24.653 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:24 smithi157 ceph-mon[33359]: pgmap v1536: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:24.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:24 smithi129 ceph-mon[30184]: pgmap v1536: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:24.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:24 smithi129 ceph-mon[42066]: pgmap v1536: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:24.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:20:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:20:25.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:25.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:25.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:26.369 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:20:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:20:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:20:26.640 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:26 smithi129 ceph-mon[30184]: pgmap v1537: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:26.640 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:26 smithi129 ceph-mon[42066]: pgmap v1537: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:26.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:26 smithi157 ceph-mon[33359]: pgmap v1537: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:28.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:28 smithi129 ceph-mon[30184]: pgmap v1538: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:28.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:28 smithi129 ceph-mon[42066]: pgmap v1538: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:28.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:28 smithi157 ceph-mon[33359]: pgmap v1538: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:30.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:30 smithi129 ceph-mon[30184]: pgmap v1539: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:30.718 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:30 smithi129 ceph-mon[42066]: pgmap v1539: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:30.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:30 smithi157 ceph-mon[33359]: pgmap v1539: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:32.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:32 smithi129 ceph-mon[30184]: pgmap v1540: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:32.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:32 smithi129 ceph-mon[42066]: pgmap v1540: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:32.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:32 smithi157 ceph-mon[33359]: pgmap v1540: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:34.664 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:34 smithi157 ceph-mon[33359]: pgmap v1541: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:34.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:34 smithi129 ceph-mon[30184]: pgmap v1541: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:34.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:34 smithi129 ceph-mon[42066]: pgmap v1541: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:34.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:20:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:20:35.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:35 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:35.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:35.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:35 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:35.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:35.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:35 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:35.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:36.391 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:20:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:20:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:20:36.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:36 smithi129 ceph-mon[42066]: pgmap v1542: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:20:36.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:36 smithi129 ceph-mon[30184]: pgmap v1542: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:20:36.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:36 smithi157 ceph-mon[33359]: pgmap v1542: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:20:38.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:38 smithi129 ceph-mon[42066]: pgmap v1543: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:38.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:38 smithi129 ceph-mon[30184]: pgmap v1543: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:38.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:38 smithi157 ceph-mon[33359]: pgmap v1543: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:40.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:40 smithi129 ceph-mon[30184]: pgmap v1544: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:20:40.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:40 smithi129 ceph-mon[42066]: pgmap v1544: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:20:40.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:40 smithi157 ceph-mon[33359]: pgmap v1544: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:20:42.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:42 smithi129 ceph-mon[30184]: pgmap v1545: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:20:42.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:42 smithi129 ceph-mon[42066]: pgmap v1545: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:20:42.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:42 smithi157 ceph-mon[33359]: pgmap v1545: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-11T21:20:44.675 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:44 smithi157 ceph-mon[33359]: pgmap v1546: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:44.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:44 smithi129 ceph-mon[30184]: pgmap v1546: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:44.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:44 smithi129 ceph-mon[42066]: pgmap v1546: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:44.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:20:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:20:45.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:45.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:45.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:46.410 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:20:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:20:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:20:46.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:46 smithi129 ceph-mon[30184]: pgmap v1547: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:46.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:46 smithi129 ceph-mon[42066]: pgmap v1547: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:46.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:46 smithi157 ceph-mon[33359]: pgmap v1547: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:48.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:48 smithi129 ceph-mon[30184]: pgmap v1548: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:48.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:48 smithi129 ceph-mon[42066]: pgmap v1548: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:48.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:48 smithi157 ceph-mon[33359]: pgmap v1548: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:50.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:50 smithi129 ceph-mon[30184]: pgmap v1549: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:50.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:50.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:50 smithi129 ceph-mon[42066]: pgmap v1549: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:50.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:50.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:50 smithi157 ceph-mon[33359]: pgmap v1549: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:50.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:20:52.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:52 smithi129 ceph-mon[30184]: pgmap v1550: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:52.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:52 smithi129 ceph-mon[42066]: pgmap v1550: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:52.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:52 smithi157 ceph-mon[33359]: pgmap v1550: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:20:54.685 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:54 smithi157 ceph-mon[33359]: pgmap v1551: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:54.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:54 smithi129 ceph-mon[30184]: pgmap v1551: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:54.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:54 smithi129 ceph-mon[42066]: pgmap v1551: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:54.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:20:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:20:55.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:55.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:55.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:20:56.429 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:20:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:20:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:20:56.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:56 smithi129 ceph-mon[30184]: pgmap v1552: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:56.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:56 smithi129 ceph-mon[42066]: pgmap v1552: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:56.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:56 smithi157 ceph-mon[33359]: pgmap v1552: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:20:58.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:20:58 smithi129 ceph-mon[30184]: pgmap v1553: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:58.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:20:58 smithi129 ceph-mon[42066]: pgmap v1553: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:20:58.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:20:58 smithi157 ceph-mon[33359]: pgmap v1553: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:00.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:00 smithi129 ceph-mon[30184]: pgmap v1554: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:00.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:00 smithi129 ceph-mon[42066]: pgmap v1554: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:00.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:00 smithi157 ceph-mon[33359]: pgmap v1554: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:02.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:02 smithi129 ceph-mon[30184]: pgmap v1555: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:02.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:02 smithi129 ceph-mon[42066]: pgmap v1555: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:02.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:02 smithi157 ceph-mon[33359]: pgmap v1555: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:04.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:04 smithi129 ceph-mon[30184]: pgmap v1556: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:04.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:04 smithi129 ceph-mon[42066]: pgmap v1556: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:04.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:21:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:21:04.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:04 smithi157 ceph-mon[33359]: pgmap v1556: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:05.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:21:05.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:05.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:21:05.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:05.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:21:05.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:06.452 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:21:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:21:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:21:06.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:06 smithi129 ceph-mon[30184]: pgmap v1557: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:06.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:06 smithi129 ceph-mon[42066]: pgmap v1557: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:06.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:06 smithi157 ceph-mon[33359]: pgmap v1557: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:08.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:08 smithi129 ceph-mon[30184]: pgmap v1558: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:08.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:08 smithi129 ceph-mon[42066]: pgmap v1558: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:08.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:08 smithi157 ceph-mon[33359]: pgmap v1558: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:10.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:10 smithi129 ceph-mon[30184]: pgmap v1559: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:10.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:10 smithi129 ceph-mon[42066]: pgmap v1559: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:10.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:10 smithi157 ceph-mon[33359]: pgmap v1559: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:12.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:12 smithi129 ceph-mon[30184]: pgmap v1560: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:12.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:12 smithi129 ceph-mon[42066]: pgmap v1560: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:12.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:12 smithi157 ceph-mon[33359]: pgmap v1560: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:14.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:14 smithi129 ceph-mon[30184]: pgmap v1561: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:14.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:14 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:14.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:14 smithi129 ceph-mon[42066]: pgmap v1561: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:14.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:14 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:14.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:21:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:21:14.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:14 smithi157 ceph-mon[33359]: pgmap v1561: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:14.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:14 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:15.564 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:15.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:15.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:16.467 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:21:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:21:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:21:16.707 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:16 smithi157 ceph-mon[33359]: pgmap v1562: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:16.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:16 smithi129 ceph-mon[30184]: pgmap v1562: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:16.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:16 smithi129 ceph-mon[42066]: pgmap v1562: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:18.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:18 smithi129 ceph-mon[30184]: pgmap v1563: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:18.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:18 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:21:18.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:18 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:21:18.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:18 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:18.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:18 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:21:18.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:18 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:21:18.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:18 smithi129 ceph-mon[42066]: pgmap v1563: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:18.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:18 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:21:18.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:18 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:21:18.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:18 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:18.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:18 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:21:18.968 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:18 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:21:18.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:18 smithi157 ceph-mon[33359]: pgmap v1563: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:18.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:18 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:21:18.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:18 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:21:18.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:18 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:18.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:18 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:21:18.975 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:18 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:21:20.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:20 smithi129 ceph-mon[30184]: pgmap v1564: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:20.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:21:20.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:20 smithi129 ceph-mon[42066]: pgmap v1564: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:20.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:21:20.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:20 smithi157 ceph-mon[33359]: pgmap v1564: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:20.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:21:22.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:22 smithi129 ceph-mon[30184]: pgmap v1565: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:22.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:22 smithi129 ceph-mon[42066]: pgmap v1565: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:22.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:22 smithi157 ceph-mon[33359]: pgmap v1565: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:24.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:24 smithi129 ceph-mon[30184]: pgmap v1566: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:24.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:24 smithi129 ceph-mon[42066]: pgmap v1566: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:24.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:24 smithi157 ceph-mon[33359]: pgmap v1566: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:24.975 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:21:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:21:25.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:25.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:25.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:26.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:21:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:21:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:21:26.744 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:26 smithi129 ceph-mon[30184]: pgmap v1567: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:26.745 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:26 smithi129 ceph-mon[42066]: pgmap v1567: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:26.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:26 smithi157 ceph-mon[33359]: pgmap v1567: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:28.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:28 smithi129 ceph-mon[30184]: pgmap v1568: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:28.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:28 smithi129 ceph-mon[42066]: pgmap v1568: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:28.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:28 smithi157 ceph-mon[33359]: pgmap v1568: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:30.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:30 smithi129 ceph-mon[30184]: pgmap v1569: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:30.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:30 smithi129 ceph-mon[42066]: pgmap v1569: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:30.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:30 smithi157 ceph-mon[33359]: pgmap v1569: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:32.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.299 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.300 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.301 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.302 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.303 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.304 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.305 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.305 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.305 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.305 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.305 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.305 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.305 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.305 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.305 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.305 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.305 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.305 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.305 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.305 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.306 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.306 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.306 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.306 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.306 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.306 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.306 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:21:32.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:32 smithi129 ceph-mon[30184]: pgmap v1570: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:32.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:32 smithi129 ceph-mon[42066]: pgmap v1570: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:32.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:32 smithi157 ceph-mon[33359]: pgmap v1570: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:34.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:34 smithi129 ceph-mon[30184]: pgmap v1571: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:34.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:34 smithi129 ceph-mon[42066]: pgmap v1571: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:34.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:34 smithi157 ceph-mon[33359]: pgmap v1571: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-11T21:21:34.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:21:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:21:35.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:35 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:21:35.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:35.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:35 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:21:35.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:35.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:35 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:21:35.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:36.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:21:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:21:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:21:36.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:36 smithi129 ceph-mon[30184]: pgmap v1572: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:36.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:36 smithi129 ceph-mon[42066]: pgmap v1572: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:36.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:36 smithi157 ceph-mon[33359]: pgmap v1572: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:38.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:38 smithi129 ceph-mon[30184]: pgmap v1573: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:38.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:38 smithi129 ceph-mon[42066]: pgmap v1573: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:38.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:38 smithi157 ceph-mon[33359]: pgmap v1573: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:40.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:40 smithi129 ceph-mon[30184]: pgmap v1574: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:40.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:40 smithi129 ceph-mon[42066]: pgmap v1574: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:40.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:40 smithi157 ceph-mon[33359]: pgmap v1574: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:42.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:42 smithi129 ceph-mon[30184]: pgmap v1575: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:42.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:42 smithi129 ceph-mon[42066]: pgmap v1575: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:42.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:42 smithi157 ceph-mon[33359]: pgmap v1575: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:44.789 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:21:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:21:44.790 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:44 smithi157 ceph-mon[33359]: pgmap v1576: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:44.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:44 smithi129 ceph-mon[30184]: pgmap v1576: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:44.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:44 smithi129 ceph-mon[42066]: pgmap v1576: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:45.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:45.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:46.467 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:21:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:21:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:21:46.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:46 smithi129 ceph-mon[30184]: pgmap v1577: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:46.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:46 smithi129 ceph-mon[42066]: pgmap v1577: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:46.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:46 smithi157 ceph-mon[33359]: pgmap v1577: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:48.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:48 smithi129 ceph-mon[30184]: pgmap v1578: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:48.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:48 smithi129 ceph-mon[42066]: pgmap v1578: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:48.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:48 smithi157 ceph-mon[33359]: pgmap v1578: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:50.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:50 smithi129 ceph-mon[30184]: pgmap v1579: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:50.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:21:50.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:50 smithi129 ceph-mon[42066]: pgmap v1579: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:50.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:21:50.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:50 smithi157 ceph-mon[33359]: pgmap v1579: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:50.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:21:52.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:52 smithi129 ceph-mon[30184]: pgmap v1580: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:52.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:52 smithi129 ceph-mon[42066]: pgmap v1580: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:52.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:52 smithi157 ceph-mon[33359]: pgmap v1580: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:21:54.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:54 smithi129 ceph-mon[30184]: pgmap v1581: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:54.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:54 smithi129 ceph-mon[42066]: pgmap v1581: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:54.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:21:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:21:54.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:54 smithi157 ceph-mon[33359]: pgmap v1581: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:55.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:55.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:55.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:21:56.467 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:21:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:21:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:21:56.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:56 smithi129 ceph-mon[30184]: pgmap v1582: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:56.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:56 smithi129 ceph-mon[42066]: pgmap v1582: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:56.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:56 smithi157 ceph-mon[33359]: pgmap v1582: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:21:58.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:21:58 smithi129 ceph-mon[30184]: pgmap v1583: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:58.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:21:58 smithi129 ceph-mon[42066]: pgmap v1583: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:21:58.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:21:58 smithi157 ceph-mon[33359]: pgmap v1583: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:00.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:00 smithi129 ceph-mon[30184]: pgmap v1584: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:00.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:00 smithi129 ceph-mon[42066]: pgmap v1584: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:00.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:00 smithi157 ceph-mon[33359]: pgmap v1584: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:02.970 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:02 smithi129 ceph-mon[30184]: pgmap v1585: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:02.971 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:02 smithi129 ceph-mon[42066]: pgmap v1585: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:02.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:02 smithi157 ceph-mon[33359]: pgmap v1585: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:04.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:04 smithi129 ceph-mon[30184]: pgmap v1586: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:04.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:04 smithi129 ceph-mon[42066]: pgmap v1586: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:04.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:22:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:22:04.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:04 smithi157 ceph-mon[33359]: pgmap v1586: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:05.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:22:05.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:05.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:22:05.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:05.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:22:05.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:06.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:22:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:22:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:22:06.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:06 smithi129 ceph-mon[30184]: pgmap v1587: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:06.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:06 smithi129 ceph-mon[42066]: pgmap v1587: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:06.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:06 smithi157 ceph-mon[33359]: pgmap v1587: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:08.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:08 smithi129 ceph-mon[30184]: pgmap v1588: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:08.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:08 smithi129 ceph-mon[42066]: pgmap v1588: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:08.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:08 smithi157 ceph-mon[33359]: pgmap v1588: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:10.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:10 smithi129 ceph-mon[30184]: pgmap v1589: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:10.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:10 smithi129 ceph-mon[42066]: pgmap v1589: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:10.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:10 smithi157 ceph-mon[33359]: pgmap v1589: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:12.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:12 smithi129 ceph-mon[30184]: pgmap v1590: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:12.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:12 smithi129 ceph-mon[42066]: pgmap v1590: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:12.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:12 smithi157 ceph-mon[33359]: pgmap v1590: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:14.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:14 smithi129 ceph-mon[30184]: pgmap v1591: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:14.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:14 smithi129 ceph-mon[42066]: pgmap v1591: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:14.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:14 smithi157 ceph-mon[33359]: pgmap v1591: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:14.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:22:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:22:15.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:15.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:15.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:16.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:22:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:22:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:22:16.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:16 smithi129 ceph-mon[30184]: pgmap v1592: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:16.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:16 smithi129 ceph-mon[42066]: pgmap v1592: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:16.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:16 smithi157 ceph-mon[33359]: pgmap v1592: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:18.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:18 smithi129 ceph-mon[30184]: pgmap v1593: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:18.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:18 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:18.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:18 smithi129 ceph-mon[42066]: pgmap v1593: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:18.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:18 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:18.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:18 smithi157 ceph-mon[33359]: pgmap v1593: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:18.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:18 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:20.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:20 smithi129 ceph-mon[42066]: pgmap v1594: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:20.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:22:20.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:22:20.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:20.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:20.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:22:20.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:22:20.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:20 smithi129 ceph-mon[30184]: pgmap v1594: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:20.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:22:20.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:22:20.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:20.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:20.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:22:20.718 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:22:20.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:20 smithi157 ceph-mon[33359]: pgmap v1594: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:20.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:22:20.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:22:20.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:20.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:20.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:22:20.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:22:22.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:22 smithi129 ceph-mon[30184]: pgmap v1595: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:22.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:22 smithi129 ceph-mon[42066]: pgmap v1595: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:22.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:22 smithi157 ceph-mon[33359]: pgmap v1595: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:24.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:24 smithi129 ceph-mon[30184]: pgmap v1596: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:24.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:24 smithi129 ceph-mon[42066]: pgmap v1596: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:24.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:24 smithi157 ceph-mon[33359]: pgmap v1596: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:25.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:22:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:22:25.716 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:25.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:25.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:26.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:22:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:22:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:22:26.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:26 smithi157 ceph-mon[33359]: pgmap v1597: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:26.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:26 smithi129 ceph-mon[30184]: pgmap v1597: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:26.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:26 smithi129 ceph-mon[42066]: pgmap v1597: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:28.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:28 smithi157 ceph-mon[33359]: pgmap v1598: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:28.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:28 smithi129 ceph-mon[30184]: pgmap v1598: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:28.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:28 smithi129 ceph-mon[42066]: pgmap v1598: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:30.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:30 smithi157 ceph-mon[33359]: pgmap v1599: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:30.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:30 smithi129 ceph-mon[42066]: pgmap v1599: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:30.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:30 smithi129 ceph-mon[30184]: pgmap v1599: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:32.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:32 smithi129 ceph-mon[30184]: pgmap v1600: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:32.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:32 smithi129 ceph-mon[42066]: pgmap v1600: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:32.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:32 smithi157 ceph-mon[33359]: pgmap v1600: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:34.787 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:34 smithi157 ceph-mon[33359]: pgmap v1601: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:34.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:34 smithi129 ceph-mon[30184]: pgmap v1601: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:34.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:34 smithi129 ceph-mon[42066]: pgmap v1601: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:35.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:22:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:22:35.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:35 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:22:35.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:35.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:35 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:22:35.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:35.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:35 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:22:35.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:36.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:22:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:22:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:22:36.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:36 smithi129 ceph-mon[30184]: pgmap v1602: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:36.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:36 smithi129 ceph-mon[42066]: pgmap v1602: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:36.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:36 smithi157 ceph-mon[33359]: pgmap v1602: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:38.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:38 smithi129 ceph-mon[30184]: pgmap v1603: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:38.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:38 smithi129 ceph-mon[42066]: pgmap v1603: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:38.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:38 smithi157 ceph-mon[33359]: pgmap v1603: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:40.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:40 smithi129 ceph-mon[30184]: pgmap v1604: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:40.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:40 smithi129 ceph-mon[42066]: pgmap v1604: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:40.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:40 smithi157 ceph-mon[33359]: pgmap v1604: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:42.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:42 smithi129 ceph-mon[30184]: pgmap v1605: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:42.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:42 smithi129 ceph-mon[42066]: pgmap v1605: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:42.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:42 smithi157 ceph-mon[33359]: pgmap v1605: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:44.776 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:44 smithi157 ceph-mon[33359]: pgmap v1606: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:44.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:44 smithi129 ceph-mon[30184]: pgmap v1606: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:44.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:44 smithi129 ceph-mon[42066]: pgmap v1606: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:45.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:22:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:22:45.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:45.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:46.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:22:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:22:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:22:46.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:46 smithi129 ceph-mon[30184]: pgmap v1607: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:46.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:46 smithi129 ceph-mon[42066]: pgmap v1607: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:46.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:46 smithi157 ceph-mon[33359]: pgmap v1607: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:48.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:48 smithi129 ceph-mon[30184]: pgmap v1608: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:48.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:48 smithi129 ceph-mon[42066]: pgmap v1608: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:48.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:48 smithi157 ceph-mon[33359]: pgmap v1608: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:50.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:50 smithi129 ceph-mon[30184]: pgmap v1609: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:50.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:22:50.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:50 smithi129 ceph-mon[42066]: pgmap v1609: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:50.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:22:50.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:50 smithi157 ceph-mon[33359]: pgmap v1609: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:50.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:22:52.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:52 smithi129 ceph-mon[30184]: pgmap v1610: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:52.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:52 smithi129 ceph-mon[42066]: pgmap v1610: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:52.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:52 smithi157 ceph-mon[33359]: pgmap v1610: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:22:54.803 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:54 smithi157 ceph-mon[33359]: pgmap v1611: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:54.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:54 smithi129 ceph-mon[30184]: pgmap v1611: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:54.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:54 smithi129 ceph-mon[42066]: pgmap v1611: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:55.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:22:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:22:55.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:55.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:55.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:22:56.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:22:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:22:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:22:56.933 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:56 smithi157 ceph-mon[33359]: pgmap v1612: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:56.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:56 smithi129 ceph-mon[42066]: pgmap v1612: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:56.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:56 smithi129 ceph-mon[30184]: pgmap v1612: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:22:57.226 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 21:22:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a[66289]: logger=cleanup t=2024-03-11T21:22:56.95005965Z level=info msg="Completed cleanup jobs" duration=3.554483ms 2024-03-11T21:22:58.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:22:58 smithi129 ceph-mon[30184]: pgmap v1613: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:58.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:22:58 smithi129 ceph-mon[42066]: pgmap v1613: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:22:58.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:22:58 smithi157 ceph-mon[33359]: pgmap v1613: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:00.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:00 smithi129 ceph-mon[30184]: pgmap v1614: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:00.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:00 smithi129 ceph-mon[42066]: pgmap v1614: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:00.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:00 smithi157 ceph-mon[33359]: pgmap v1614: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:02.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:02 smithi129 ceph-mon[30184]: pgmap v1615: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:02.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:02 smithi129 ceph-mon[42066]: pgmap v1615: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:02.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:02 smithi157 ceph-mon[33359]: pgmap v1615: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:04.803 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:04 smithi157 ceph-mon[33359]: pgmap v1616: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:04.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:04 smithi129 ceph-mon[30184]: pgmap v1616: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:04.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:04 smithi129 ceph-mon[42066]: pgmap v1616: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:05.224 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:23:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:23:05.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:23:05.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:05.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:23:05.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:05.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:23:05.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:06.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:23:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:23:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:23:06.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:06 smithi129 ceph-mon[30184]: pgmap v1617: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:06.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:06 smithi129 ceph-mon[42066]: pgmap v1617: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:06.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:06 smithi157 ceph-mon[33359]: pgmap v1617: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:08.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:08 smithi129 ceph-mon[30184]: pgmap v1618: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:08.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:08 smithi129 ceph-mon[42066]: pgmap v1618: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:08.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:08 smithi157 ceph-mon[33359]: pgmap v1618: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:10.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:10 smithi129 ceph-mon[30184]: pgmap v1619: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:10.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:10 smithi129 ceph-mon[42066]: pgmap v1619: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:10.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:10 smithi157 ceph-mon[33359]: pgmap v1619: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:12.310 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.311 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.312 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.313 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.314 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.315 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.316 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:23:12.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:12 smithi129 ceph-mon[30184]: pgmap v1620: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:12.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:12 smithi129 ceph-mon[42066]: pgmap v1620: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:12.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:12 smithi157 ceph-mon[33359]: pgmap v1620: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:14.815 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:14 smithi157 ceph-mon[33359]: pgmap v1621: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:14.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:14 smithi129 ceph-mon[30184]: pgmap v1621: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:14.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:14 smithi129 ceph-mon[42066]: pgmap v1621: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:15.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:23:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:23:15.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:15.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:15.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:16.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:23:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:23:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:23:16.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:16 smithi129 ceph-mon[30184]: pgmap v1622: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:16.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:16 smithi129 ceph-mon[42066]: pgmap v1622: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:16.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:16 smithi157 ceph-mon[33359]: pgmap v1622: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:18.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:18 smithi129 ceph-mon[30184]: pgmap v1623: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:18.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:18 smithi129 ceph-mon[42066]: pgmap v1623: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:18.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:18 smithi157 ceph-mon[33359]: pgmap v1623: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:19.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:19 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:19.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:19 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:19.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:19 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:20.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:20 smithi129 ceph-mon[30184]: pgmap v1624: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:20.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:23:20.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:20 smithi129 ceph-mon[42066]: pgmap v1624: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:20.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:23:20.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:20 smithi157 ceph-mon[33359]: pgmap v1624: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:20.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:23:21.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:21 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:23:21.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:21 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:23:21.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:21 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:21.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:21 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:21.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:21 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:23:21.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:21 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:23:21.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:21 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:23:21.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:21 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:21.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:21 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:21.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:21 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:23:21.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:21 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:23:21.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:21 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:23:21.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:21 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:21.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:21 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:21.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:21 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:23:22.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:22 smithi129 ceph-mon[30184]: pgmap v1625: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:22.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:22 smithi129 ceph-mon[42066]: pgmap v1625: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:22.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:22 smithi157 ceph-mon[33359]: pgmap v1625: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:24.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:24 smithi129 ceph-mon[30184]: pgmap v1626: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:24.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:24 smithi129 ceph-mon[42066]: pgmap v1626: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:24.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:23:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:23:24.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:24 smithi157 ceph-mon[33359]: pgmap v1626: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:25.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:25.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:25.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:26.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:23:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:23:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:23:26.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:26 smithi129 ceph-mon[30184]: pgmap v1627: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:26.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:26 smithi129 ceph-mon[42066]: pgmap v1627: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:26.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:26 smithi157 ceph-mon[33359]: pgmap v1627: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:28.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:28 smithi129 ceph-mon[30184]: pgmap v1628: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:28.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:28 smithi129 ceph-mon[42066]: pgmap v1628: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:28.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:28 smithi157 ceph-mon[33359]: pgmap v1628: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:30.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:30 smithi129 ceph-mon[30184]: pgmap v1629: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:30.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:30 smithi129 ceph-mon[42066]: pgmap v1629: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:30.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:30 smithi157 ceph-mon[33359]: pgmap v1629: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:32.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:32 smithi129 ceph-mon[30184]: pgmap v1630: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:32.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:32 smithi129 ceph-mon[42066]: pgmap v1630: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:32.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:32 smithi157 ceph-mon[33359]: pgmap v1630: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:34.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:34 smithi129 ceph-mon[30184]: pgmap v1631: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:34.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:34 smithi129 ceph-mon[42066]: pgmap v1631: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:34.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:23:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:23:34.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:34 smithi157 ceph-mon[33359]: pgmap v1631: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:35.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:35 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:23:35.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:35.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:35 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:23:35.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:35.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:35 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:23:35.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:36.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:23:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:23:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:23:36.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:36 smithi129 ceph-mon[30184]: pgmap v1632: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:36.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:36 smithi129 ceph-mon[42066]: pgmap v1632: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:36.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:36 smithi157 ceph-mon[33359]: pgmap v1632: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:38.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:38 smithi129 ceph-mon[30184]: pgmap v1633: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:38.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:38 smithi129 ceph-mon[42066]: pgmap v1633: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:38.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:38 smithi157 ceph-mon[33359]: pgmap v1633: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:40.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:40 smithi129 ceph-mon[30184]: pgmap v1634: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:40.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:40 smithi129 ceph-mon[42066]: pgmap v1634: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:40.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:40 smithi157 ceph-mon[33359]: pgmap v1634: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:42.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:42 smithi129 ceph-mon[30184]: pgmap v1635: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:42.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:42 smithi129 ceph-mon[42066]: pgmap v1635: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:42.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:42 smithi157 ceph-mon[33359]: pgmap v1635: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:44.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:44 smithi129 ceph-mon[30184]: pgmap v1636: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:44.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:44 smithi129 ceph-mon[42066]: pgmap v1636: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:44.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:44 smithi157 ceph-mon[33359]: pgmap v1636: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:44.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:23:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:23:45.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:45.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:45.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:46.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:23:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:23:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:23:46.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:46 smithi129 ceph-mon[30184]: pgmap v1637: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:46.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:46 smithi129 ceph-mon[42066]: pgmap v1637: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:46.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:46 smithi157 ceph-mon[33359]: pgmap v1637: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:48.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:48 smithi129 ceph-mon[30184]: pgmap v1638: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:48.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:48 smithi129 ceph-mon[42066]: pgmap v1638: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:48.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:48 smithi157 ceph-mon[33359]: pgmap v1638: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:50.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:50 smithi129 ceph-mon[30184]: pgmap v1639: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:50.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:23:50.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:50 smithi129 ceph-mon[42066]: pgmap v1639: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:50.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:23:50.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:50 smithi157 ceph-mon[33359]: pgmap v1639: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:50.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:23:52.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:52 smithi129 ceph-mon[30184]: pgmap v1640: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:52.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:52 smithi129 ceph-mon[42066]: pgmap v1640: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:52.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:52 smithi157 ceph-mon[33359]: pgmap v1640: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:23:54.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:54 smithi129 ceph-mon[30184]: pgmap v1641: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:54.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:54 smithi129 ceph-mon[42066]: pgmap v1641: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:54.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:54 smithi157 ceph-mon[33359]: pgmap v1641: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:54.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:23:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:23:55.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:55.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:55.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:23:56.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:23:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:23:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:23:56.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:56 smithi129 ceph-mon[30184]: pgmap v1642: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:56.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:56 smithi129 ceph-mon[42066]: pgmap v1642: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:56.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:56 smithi157 ceph-mon[33359]: pgmap v1642: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:23:58.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:23:58 smithi129 ceph-mon[30184]: pgmap v1643: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:58.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:23:58 smithi129 ceph-mon[42066]: pgmap v1643: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:23:58.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:23:58 smithi157 ceph-mon[33359]: pgmap v1643: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:00.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:00 smithi129 ceph-mon[30184]: pgmap v1644: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:00.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:00 smithi129 ceph-mon[42066]: pgmap v1644: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:00.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:00 smithi157 ceph-mon[33359]: pgmap v1644: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:02.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:02 smithi129 ceph-mon[30184]: pgmap v1645: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:24:02.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:02 smithi129 ceph-mon[42066]: pgmap v1645: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:24:02.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:02 smithi157 ceph-mon[33359]: pgmap v1645: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:24:04.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:24:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:24:04.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:04 smithi157 ceph-mon[33359]: pgmap v1646: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:05.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:04 smithi129 ceph-mon[30184]: pgmap v1646: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:05.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:04 smithi129 ceph-mon[42066]: pgmap v1646: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:05.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.317 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.318 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.319 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.320 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.321 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.321 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.321 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.321 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.321 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.321 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.321 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.321 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.321 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.321 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 1 sec 2024-03-11T21:24:05.321 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: started 512 aios 2024-03-11T21:24:05.321 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x5582546ed600 2024-03-11T21:24:05.321 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a2dc20 2024-03-11T21:24:05.322 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a2e5c0 2024-03-11T21:24:05.322 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a2ebc0 2024-03-11T21:24:05.322 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a2eab0 2024-03-11T21:24:05.322 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a2f110 2024-03-11T21:24:05.322 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254996810 2024-03-11T21:24:05.322 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a2f9b0 2024-03-11T21:24:05.322 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a2fec0 2024-03-11T21:24:05.322 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a30370 2024-03-11T21:24:05.322 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a30020 2024-03-11T21:24:05.322 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a30150 2024-03-11T21:24:05.322 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a30260 2024-03-11T21:24:05.322 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a30990 2024-03-11T21:24:05.322 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a2fac0 2024-03-11T21:24:05.323 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a2fbf0 2024-03-11T21:24:05.323 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a2fd00 2024-03-11T21:24:05.323 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a30e60 2024-03-11T21:24:05.323 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a304e0 2024-03-11T21:24:05.323 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a30aa0 2024-03-11T21:24:05.323 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a30bd0 2024-03-11T21:24:05.323 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a30ce0 2024-03-11T21:24:05.323 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a31420 2024-03-11T21:24:05.323 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a31310 2024-03-11T21:24:05.323 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a30630 2024-03-11T21:24:05.323 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a30740 2024-03-11T21:24:05.323 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a30850 2024-03-11T21:24:05.323 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a31a20 2024-03-11T21:24:05.324 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a31530 2024-03-11T21:24:05.324 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a30f70 2024-03-11T21:24:05.324 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a310a0 2024-03-11T21:24:05.324 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a311d0 2024-03-11T21:24:05.324 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a31fe0 2024-03-11T21:24:05.324 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a31b30 2024-03-11T21:24:05.324 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a31c60 2024-03-11T21:24:05.324 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a31d90 2024-03-11T21:24:05.324 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a31ec0 2024-03-11T21:24:05.324 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a325c0 2024-03-11T21:24:05.324 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a324b0 2024-03-11T21:24:05.324 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a31680 2024-03-11T21:24:05.324 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a317b0 2024-03-11T21:24:05.324 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a326d0 2024-03-11T21:24:05.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a327e0 2024-03-11T21:24:05.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a32de0 2024-03-11T21:24:05.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a32930 2024-03-11T21:24:05.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a32a60 2024-03-11T21:24:05.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a32bb0 2024-03-11T21:24:05.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a32ef0 2024-03-11T21:24:05.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a333a0 2024-03-11T21:24:05.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a334b0 2024-03-11T21:24:05.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a33a70 2024-03-11T21:24:05.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a33000 2024-03-11T21:24:05.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a33110 2024-03-11T21:24:05.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a33240 2024-03-11T21:24:05.325 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a33bc0 2024-03-11T21:24:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a335c0 2024-03-11T21:24:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a33710 2024-03-11T21:24:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a33840 2024-03-11T21:24:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a33cd0 2024-03-11T21:24:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a34180 2024-03-11T21:24:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a342b0 2024-03-11T21:24:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a34890 2024-03-11T21:24:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a33de0 2024-03-11T21:24:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a33ef0 2024-03-11T21:24:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a34000 2024-03-11T21:24:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a349e0 2024-03-11T21:24:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a343e0 2024-03-11T21:24:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a32110 2024-03-11T21:24:05.326 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a32220 2024-03-11T21:24:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a32350 2024-03-11T21:24:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a34fe0 2024-03-11T21:24:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a34b30 2024-03-11T21:24:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a344f0 2024-03-11T21:24:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a34640 2024-03-11T21:24:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a34770 2024-03-11T21:24:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a355e0 2024-03-11T21:24:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a35130 2024-03-11T21:24:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a35280 2024-03-11T21:24:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a353b0 2024-03-11T21:24:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a356f0 2024-03-11T21:24:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a35ba0 2024-03-11T21:24:05.327 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a35cb0 2024-03-11T21:24:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a36270 2024-03-11T21:24:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a35800 2024-03-11T21:24:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a35910 2024-03-11T21:24:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a35a60 2024-03-11T21:24:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a36380 2024-03-11T21:24:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a35dc0 2024-03-11T21:24:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a35ef0 2024-03-11T21:24:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a36020 2024-03-11T21:24:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a36150 2024-03-11T21:24:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a36980 2024-03-11T21:24:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a364d0 2024-03-11T21:24:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a34c40 2024-03-11T21:24:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a34d50 2024-03-11T21:24:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a34ea0 2024-03-11T21:24:05.328 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a36f80 2024-03-11T21:24:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a36ad0 2024-03-11T21:24:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a365e0 2024-03-11T21:24:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a36710 2024-03-11T21:24:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a36820 2024-03-11T21:24:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a37580 2024-03-11T21:24:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a370d0 2024-03-11T21:24:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a36be0 2024-03-11T21:24:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a36d10 2024-03-11T21:24:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a36e40 2024-03-11T21:24:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a37b80 2024-03-11T21:24:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a376d0 2024-03-11T21:24:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a37cb0 2024-03-11T21:24:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a37de0 2024-03-11T21:24:05.329 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a383c0 2024-03-11T21:24:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a377e0 2024-03-11T21:24:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a37910 2024-03-11T21:24:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a37a40 2024-03-11T21:24:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a384d0 2024-03-11T21:24:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a37f10 2024-03-11T21:24:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a38040 2024-03-11T21:24:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a38150 2024-03-11T21:24:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a38280 2024-03-11T21:24:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a38ad0 2024-03-11T21:24:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a38620 2024-03-11T21:24:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a371e0 2024-03-11T21:24:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a372f0 2024-03-11T21:24:05.330 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a37440 2024-03-11T21:24:05.331 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a390d0 2024-03-11T21:24:05.331 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a38c20 2024-03-11T21:24:05.331 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a38d50 2024-03-11T21:24:05.331 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a38e60 2024-03-11T21:24:05.331 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a38fb0 2024-03-11T21:24:05.331 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a396d0 2024-03-11T21:24:05.331 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a39220 2024-03-11T21:24:05.331 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a39350 2024-03-11T21:24:05.331 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a39460 2024-03-11T21:24:05.331 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a395b0 2024-03-11T21:24:05.331 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a39cd0 2024-03-11T21:24:05.331 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a39820 2024-03-11T21:24:05.331 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a39950 2024-03-11T21:24:05.331 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a39a60 2024-03-11T21:24:05.332 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a39bb0 2024-03-11T21:24:05.332 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3a2d0 2024-03-11T21:24:05.332 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a39e20 2024-03-11T21:24:05.332 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a39f70 2024-03-11T21:24:05.332 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3a080 2024-03-11T21:24:05.332 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3a420 2024-03-11T21:24:05.332 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3a8f0 2024-03-11T21:24:05.332 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3aa40 2024-03-11T21:24:05.332 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3a530 2024-03-11T21:24:05.332 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3a680 2024-03-11T21:24:05.332 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3a7d0 2024-03-11T21:24:05.332 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3ab70 2024-03-11T21:24:05.332 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3b060 2024-03-11T21:24:05.332 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3b170 2024-03-11T21:24:05.333 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3b730 2024-03-11T21:24:05.333 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3acc0 2024-03-11T21:24:05.333 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3adf0 2024-03-11T21:24:05.333 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3af40 2024-03-11T21:24:05.333 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3b880 2024-03-11T21:24:05.333 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3b280 2024-03-11T21:24:05.333 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3b3b0 2024-03-11T21:24:05.333 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3b4e0 2024-03-11T21:24:05.333 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3b610 2024-03-11T21:24:05.333 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3be80 2024-03-11T21:24:05.333 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3b9d0 2024-03-11T21:24:05.333 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3bb00 2024-03-11T21:24:05.333 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3bc10 2024-03-11T21:24:05.333 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3bd60 2024-03-11T21:24:05.334 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3c460 2024-03-11T21:24:05.334 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3bfb0 2024-03-11T21:24:05.334 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3c100 2024-03-11T21:24:05.334 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3c230 2024-03-11T21:24:05.334 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3c5b0 2024-03-11T21:24:05.334 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3caa0 2024-03-11T21:24:05.334 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3cbb0 2024-03-11T21:24:05.334 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3d190 2024-03-11T21:24:05.334 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3c700 2024-03-11T21:24:05.334 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3c830 2024-03-11T21:24:05.334 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3c960 2024-03-11T21:24:05.334 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3d2a0 2024-03-11T21:24:05.334 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3cce0 2024-03-11T21:24:05.334 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3ce30 2024-03-11T21:24:05.335 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3cf60 2024-03-11T21:24:05.335 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3d3b0 2024-03-11T21:24:05.335 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3d880 2024-03-11T21:24:05.335 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3d9f0 2024-03-11T21:24:05.335 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3d070 2024-03-11T21:24:05.335 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3db20 2024-03-11T21:24:05.335 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3dc50 2024-03-11T21:24:05.335 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3d4c0 2024-03-11T21:24:05.335 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3d5d0 2024-03-11T21:24:05.335 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3d720 2024-03-11T21:24:05.335 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3dd80 2024-03-11T21:24:05.335 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3df00 2024-03-11T21:24:05.335 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3e500 2024-03-11T21:24:05.335 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3e050 2024-03-11T21:24:05.335 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3e1a0 2024-03-11T21:24:05.336 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3e310 2024-03-11T21:24:05.336 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3e610 2024-03-11T21:24:05.336 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3eb00 2024-03-11T21:24:05.336 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3ec30 2024-03-11T21:24:05.336 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a38730 2024-03-11T21:24:05.336 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a38860 2024-03-11T21:24:05.336 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a38970 2024-03-11T21:24:05.336 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3ed60 2024-03-11T21:24:05.336 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3f210 2024-03-11T21:24:05.336 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3f360 2024-03-11T21:24:05.336 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3f940 2024-03-11T21:24:05.336 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3ee70 2024-03-11T21:24:05.336 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3efa0 2024-03-11T21:24:05.336 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3f0d0 2024-03-11T21:24:05.337 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3fa70 2024-03-11T21:24:05.337 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3f490 2024-03-11T21:24:05.337 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3f5c0 2024-03-11T21:24:05.337 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3f6d0 2024-03-11T21:24:05.337 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3f800 2024-03-11T21:24:05.337 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a40050 2024-03-11T21:24:05.337 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3fba0 2024-03-11T21:24:05.337 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3fcf0 2024-03-11T21:24:05.337 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3fe00 2024-03-11T21:24:05.337 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3ff10 2024-03-11T21:24:05.337 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a40650 2024-03-11T21:24:05.337 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a401a0 2024-03-11T21:24:05.338 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a402f0 2024-03-11T21:24:05.338 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a40420 2024-03-11T21:24:05.338 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a407c0 2024-03-11T21:24:05.338 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a40930 2024-03-11T21:24:05.338 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a40f30 2024-03-11T21:24:05.338 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a40a80 2024-03-11T21:24:05.338 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a40bb0 2024-03-11T21:24:05.338 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a40cc0 2024-03-11T21:24:05.338 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a40df0 2024-03-11T21:24:05.338 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a41530 2024-03-11T21:24:05.338 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a41080 2024-03-11T21:24:05.338 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a40530 2024-03-11T21:24:05.339 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a41190 2024-03-11T21:24:05.339 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a412c0 2024-03-11T21:24:05.339 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a41b10 2024-03-11T21:24:05.339 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a41660 2024-03-11T21:24:05.339 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a41790 2024-03-11T21:24:05.339 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a418e0 2024-03-11T21:24:05.339 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a41c60 2024-03-11T21:24:05.339 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a42130 2024-03-11T21:24:05.339 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a42260 2024-03-11T21:24:05.339 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3e760 2024-03-11T21:24:05.339 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3e890 2024-03-11T21:24:05.339 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a3e9a0 2024-03-11T21:24:05.339 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a42370 2024-03-11T21:24:05.339 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a42860 2024-03-11T21:24:05.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a42990 2024-03-11T21:24:05.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a42f50 2024-03-11T21:24:05.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a424c0 2024-03-11T21:24:05.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a425d0 2024-03-11T21:24:05.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a42720 2024-03-11T21:24:05.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a430a0 2024-03-11T21:24:05.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a42aa0 2024-03-11T21:24:05.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a42bd0 2024-03-11T21:24:05.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a42d40 2024-03-11T21:24:05.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a431b0 2024-03-11T21:24:05.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a43680 2024-03-11T21:24:05.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a437b0 2024-03-11T21:24:05.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a41d90 2024-03-11T21:24:05.340 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a41ec0 2024-03-11T21:24:05.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a41ff0 2024-03-11T21:24:05.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a438e0 2024-03-11T21:24:05.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a43d90 2024-03-11T21:24:05.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a43ec0 2024-03-11T21:24:05.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a432e0 2024-03-11T21:24:05.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a43430 2024-03-11T21:24:05.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a43540 2024-03-11T21:24:05.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a43ff0 2024-03-11T21:24:05.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a444c0 2024-03-11T21:24:05.341 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a44610 2024-03-11T21:24:05.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a44100 2024-03-11T21:24:05.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a44250 2024-03-11T21:24:05.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a443a0 2024-03-11T21:24:05.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a44720 2024-03-11T21:24:05.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a44890 2024-03-11T21:24:05.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a44e90 2024-03-11T21:24:05.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a449e0 2024-03-11T21:24:05.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a44b30 2024-03-11T21:24:05.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a44c80 2024-03-11T21:24:05.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a44fc0 2024-03-11T21:24:05.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a454d0 2024-03-11T21:24:05.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a45620 2024-03-11T21:24:05.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a439f0 2024-03-11T21:24:05.342 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a43b20 2024-03-11T21:24:05.343 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a43c70 2024-03-11T21:24:05.343 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a45730 2024-03-11T21:24:05.343 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a45880 2024-03-11T21:24:05.343 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a45e80 2024-03-11T21:24:05.343 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a459d0 2024-03-11T21:24:05.343 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a45b00 2024-03-11T21:24:05.343 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a45c50 2024-03-11T21:24:05.343 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a45fb0 2024-03-11T21:24:05.343 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a464d0 2024-03-11T21:24:05.343 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a46600 2024-03-11T21:24:05.343 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a45d60 2024-03-11T21:24:05.343 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a46730 2024-03-11T21:24:05.343 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a46880 2024-03-11T21:24:05.343 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a460e0 2024-03-11T21:24:05.344 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a46210 2024-03-11T21:24:05.344 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a46340 2024-03-11T21:24:05.344 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a469d0 2024-03-11T21:24:05.344 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a46ea0 2024-03-11T21:24:05.344 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a46fd0 2024-03-11T21:24:05.344 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a45130 2024-03-11T21:24:05.344 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a45260 2024-03-11T21:24:05.344 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a45390 2024-03-11T21:24:05.344 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a47100 2024-03-11T21:24:05.344 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a475d0 2024-03-11T21:24:05.344 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a47720 2024-03-11T21:24:05.344 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a46b00 2024-03-11T21:24:05.344 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a46c30 2024-03-11T21:24:05.344 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a46d60 2024-03-11T21:24:05.345 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a47850 2024-03-11T21:24:05.345 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a47d20 2024-03-11T21:24:05.345 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a47e50 2024-03-11T21:24:05.345 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a48410 2024-03-11T21:24:05.345 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a47980 2024-03-11T21:24:05.345 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a47ab0 2024-03-11T21:24:05.345 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a47c00 2024-03-11T21:24:05.345 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a48560 2024-03-11T21:24:05.345 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a47f60 2024-03-11T21:24:05.345 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a48090 2024-03-11T21:24:05.345 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a481a0 2024-03-11T21:24:05.345 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a482f0 2024-03-11T21:24:05.345 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a48b40 2024-03-11T21:24:05.346 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a48690 2024-03-11T21:24:05.346 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a487c0 2024-03-11T21:24:05.346 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a488d0 2024-03-11T21:24:05.346 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a48a20 2024-03-11T21:24:05.346 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a49140 2024-03-11T21:24:05.346 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a48c90 2024-03-11T21:24:05.346 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a48de0 2024-03-11T21:24:05.346 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a48f30 2024-03-11T21:24:05.346 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a492b0 2024-03-11T21:24:05.346 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a49760 2024-03-11T21:24:05.346 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a49890 2024-03-11T21:24:05.346 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a47230 2024-03-11T21:24:05.346 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a47360 2024-03-11T21:24:05.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a47490 2024-03-11T21:24:05.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a499a0 2024-03-11T21:24:05.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a49e70 2024-03-11T21:24:05.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a49fc0 2024-03-11T21:24:05.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a493c0 2024-03-11T21:24:05.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a494f0 2024-03-11T21:24:05.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a49620 2024-03-11T21:24:05.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4a0f0 2024-03-11T21:24:05.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4a5c0 2024-03-11T21:24:05.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4a710 2024-03-11T21:24:05.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a49ad0 2024-03-11T21:24:05.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a49c00 2024-03-11T21:24:05.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a49d30 2024-03-11T21:24:05.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4a840 2024-03-11T21:24:05.347 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4ad10 2024-03-11T21:24:05.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4ae60 2024-03-11T21:24:05.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4a220 2024-03-11T21:24:05.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4a350 2024-03-11T21:24:05.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4a460 2024-03-11T21:24:05.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4af90 2024-03-11T21:24:05.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4b460 2024-03-11T21:24:05.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4b590 2024-03-11T21:24:05.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4bb70 2024-03-11T21:24:05.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4b0c0 2024-03-11T21:24:05.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4b1f0 2024-03-11T21:24:05.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4b320 2024-03-11T21:24:05.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4bcc0 2024-03-11T21:24:05.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4b6c0 2024-03-11T21:24:05.348 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4b810 2024-03-11T21:24:05.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4b920 2024-03-11T21:24:05.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4bdf0 2024-03-11T21:24:05.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4c310 2024-03-11T21:24:05.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4c490 2024-03-11T21:24:05.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4a970 2024-03-11T21:24:05.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4aac0 2024-03-11T21:24:05.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4abf0 2024-03-11T21:24:05.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4c5a0 2024-03-11T21:24:05.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4ca50 2024-03-11T21:24:05.349 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4cba0 2024-03-11T21:24:05.350 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4bf20 2024-03-11T21:24:05.350 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4c050 2024-03-11T21:24:05.350 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4c180 2024-03-11T21:24:05.350 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4ccd0 2024-03-11T21:24:05.350 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4d180 2024-03-11T21:24:05.350 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4d290 2024-03-11T21:24:05.350 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4d870 2024-03-11T21:24:05.350 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4cde0 2024-03-11T21:24:05.350 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4cf30 2024-03-11T21:24:05.350 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4d3c0 2024-03-11T21:24:05.350 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4d4f0 2024-03-11T21:24:05.350 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4d620 2024-03-11T21:24:05.350 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4d750 2024-03-11T21:24:05.351 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4de70 2024-03-11T21:24:05.351 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4d9c0 2024-03-11T21:24:05.351 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4db10 2024-03-11T21:24:05.351 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4dc40 2024-03-11T21:24:05.351 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4dfa0 2024-03-11T21:24:05.351 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4e470 2024-03-11T21:24:05.351 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4e5a0 2024-03-11T21:24:05.351 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4c6b0 2024-03-11T21:24:05.351 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4c7e0 2024-03-11T21:24:05.351 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4c910 2024-03-11T21:24:05.351 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4e6d0 2024-03-11T21:24:05.351 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4eba0 2024-03-11T21:24:05.351 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4ecf0 2024-03-11T21:24:05.351 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4e0d0 2024-03-11T21:24:05.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4e200 2024-03-11T21:24:05.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4e330 2024-03-11T21:24:05.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4ee20 2024-03-11T21:24:05.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4f2f0 2024-03-11T21:24:05.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4f440 2024-03-11T21:24:05.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4dd50 2024-03-11T21:24:05.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4f910 2024-03-11T21:24:05.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4f570 2024-03-11T21:24:05.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4f6c0 2024-03-11T21:24:05.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4ef30 2024-03-11T21:24:05.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4f080 2024-03-11T21:24:05.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4f1b0 2024-03-11T21:24:05.352 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4fed0 2024-03-11T21:24:05.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4fa20 2024-03-11T21:24:05.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4fb70 2024-03-11T21:24:05.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4fca0 2024-03-11T21:24:05.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a50000 2024-03-11T21:24:05.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a504d0 2024-03-11T21:24:05.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a50600 2024-03-11T21:24:05.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4fdb0 2024-03-11T21:24:05.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a50730 2024-03-11T21:24:05.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a50880 2024-03-11T21:24:05.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a50110 2024-03-11T21:24:05.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a50260 2024-03-11T21:24:05.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a50390 2024-03-11T21:24:05.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a509d0 2024-03-11T21:24:05.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a50ea0 2024-03-11T21:24:05.353 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a50fd0 2024-03-11T21:24:05.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4e800 2024-03-11T21:24:05.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4e950 2024-03-11T21:24:05.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a4ea80 2024-03-11T21:24:05.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a510e0 2024-03-11T21:24:05.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a515b0 2024-03-11T21:24:05.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a516e0 2024-03-11T21:24:05.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a50b00 2024-03-11T21:24:05.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a50c50 2024-03-11T21:24:05.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a50d80 2024-03-11T21:24:05.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a517f0 2024-03-11T21:24:05.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a51960 2024-03-11T21:24:05.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a51f60 2024-03-11T21:24:05.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a51ab0 2024-03-11T21:24:05.354 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a51be0 2024-03-11T21:24:06.092 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:06.092 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:06.093 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:06.093 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:06.224 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:06.224 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:06.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:24:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:24:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:24:07.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:06 smithi129 ceph-mon[30184]: pgmap v1647: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:07.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:06 smithi129 ceph-mon[30184]: osdmap e755: 8 total, 8 up, 8 in 2024-03-11T21:24:07.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:06 smithi129 ceph-mon[42066]: pgmap v1647: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:07.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:06 smithi129 ceph-mon[42066]: osdmap e755: 8 total, 8 up, 8 in 2024-03-11T21:24:07.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:06 smithi157 ceph-mon[33359]: pgmap v1647: 228 pgs: 228 active+clean; 582 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:07.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:06 smithi157 ceph-mon[33359]: osdmap e755: 8 total, 8 up, 8 in 2024-03-11T21:24:08.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:07 smithi129 ceph-mon[30184]: osdmap e756: 8 total, 8 up, 8 in 2024-03-11T21:24:08.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:07 smithi129 ceph-mon[42066]: osdmap e756: 8 total, 8 up, 8 in 2024-03-11T21:24:08.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:07 smithi157 ceph-mon[33359]: osdmap e756: 8 total, 8 up, 8 in 2024-03-11T21:24:09.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:08 smithi129 ceph-mon[30184]: pgmap v1650: 164 pgs: 164 active+clean; 582 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:09.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:08 smithi129 ceph-mon[30184]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:24:09.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:08 smithi129 ceph-mon[30184]: osdmap e757: 8 total, 8 up, 8 in 2024-03-11T21:24:09.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:24:09.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:08 smithi129 ceph-mon[42066]: pgmap v1650: 164 pgs: 164 active+clean; 582 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:09.218 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:08 smithi129 ceph-mon[42066]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:24:09.218 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:08 smithi129 ceph-mon[42066]: osdmap e757: 8 total, 8 up, 8 in 2024-03-11T21:24:09.218 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:24:09.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:08 smithi157 ceph-mon[33359]: pgmap v1650: 164 pgs: 164 active+clean; 582 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:09.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:08 smithi157 ceph-mon[33359]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:24:09.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:08 smithi157 ceph-mon[33359]: osdmap e757: 8 total, 8 up, 8 in 2024-03-11T21:24:09.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:24:10.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:24:10.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:09 smithi129 ceph-mon[30184]: osdmap e758: 8 total, 8 up, 8 in 2024-03-11T21:24:10.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-11T21:24:10.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:24:10.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:24:10.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:09 smithi129 ceph-mon[42066]: osdmap e758: 8 total, 8 up, 8 in 2024-03-11T21:24:10.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-11T21:24:10.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:24:10.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-11T21:24:10.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:09 smithi157 ceph-mon[33359]: osdmap e758: 8 total, 8 up, 8 in 2024-03-11T21:24:10.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-11T21:24:10.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:24:11.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:10 smithi129 ceph-mon[30184]: pgmap v1653: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:11.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-11T21:24:11.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:10 smithi129 ceph-mon[30184]: osdmap e759: 8 total, 8 up, 8 in 2024-03-11T21:24:11.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "app2"}]: dispatch 2024-03-11T21:24:11.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:24:11.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:10 smithi129 ceph-mon[42066]: pgmap v1653: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:11.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-11T21:24:11.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:10 smithi129 ceph-mon[42066]: osdmap e759: 8 total, 8 up, 8 in 2024-03-11T21:24:11.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "app2"}]: dispatch 2024-03-11T21:24:11.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:24:11.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:10 smithi157 ceph-mon[33359]: pgmap v1653: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:11.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-11T21:24:11.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:10 smithi157 ceph-mon[33359]: osdmap e759: 8 total, 8 up, 8 in 2024-03-11T21:24:11.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "app2"}]: dispatch 2024-03-11T21:24:11.224 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:24:12.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-11T21:24:12.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:11 smithi129 ceph-mon[30184]: osdmap e760: 8 total, 8 up, 8 in 2024-03-11T21:24:12.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi129-82761-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-11T21:24:12.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:11 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:24:12.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-11T21:24:12.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:11 smithi129 ceph-mon[42066]: osdmap e760: 8 total, 8 up, 8 in 2024-03-11T21:24:12.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi129-82761-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-11T21:24:12.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:11 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:24:12.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi129-82761-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-11T21:24:12.224 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:11 smithi157 ceph-mon[33359]: osdmap e760: 8 total, 8 up, 8 in 2024-03-11T21:24:12.224 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi129-82761-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-11T21:24:12.224 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:11 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-11T21:24:13.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:12 smithi129 ceph-mon[30184]: pgmap v1656: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 873 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:24:13.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-11T21:24:13.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:12 smithi129 ceph-mon[30184]: osdmap e761: 8 total, 8 up, 8 in 2024-03-11T21:24:13.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T21:24:13.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-11T21:24:13.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:12 smithi129 ceph-mon[42066]: pgmap v1656: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 873 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:24:13.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-11T21:24:13.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:12 smithi129 ceph-mon[42066]: osdmap e761: 8 total, 8 up, 8 in 2024-03-11T21:24:13.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T21:24:13.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-11T21:24:13.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:12 smithi157 ceph-mon[33359]: pgmap v1656: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 873 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:24:13.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-11T21:24:13.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:12 smithi157 ceph-mon[33359]: osdmap e761: 8 total, 8 up, 8 in 2024-03-11T21:24:13.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-11T21:24:13.224 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-11T21:24:13.793 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a51d10 2024-03-11T21:24:13.793 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a51e40 2024-03-11T21:24:13.793 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a52540 2024-03-11T21:24:13.793 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a52090 2024-03-11T21:24:13.794 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a521e0 2024-03-11T21:24:13.794 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a52310 2024-03-11T21:24:13.794 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a52670 2024-03-11T21:24:13.794 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a52b90 2024-03-11T21:24:13.794 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a52cc0 2024-03-11T21:24:13.794 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a511f0 2024-03-11T21:24:13.794 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a51320 2024-03-11T21:24:13.794 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a51450 2024-03-11T21:24:13.794 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a52df0 2024-03-11T21:24:13.794 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a532c0 2024-03-11T21:24:13.794 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a533f0 2024-03-11T21:24:13.795 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a527a0 2024-03-11T21:24:13.795 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a528f0 2024-03-11T21:24:13.795 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a52a00 2024-03-11T21:24:13.795 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a53500 2024-03-11T21:24:13.795 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a539d0 2024-03-11T21:24:13.795 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a53b20 2024-03-11T21:24:13.795 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a52f00 2024-03-11T21:24:13.795 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a53030 2024-03-11T21:24:13.795 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a53160 2024-03-11T21:24:13.795 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a53c50 2024-03-11T21:24:13.795 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a54120 2024-03-11T21:24:13.795 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a54250 2024-03-11T21:24:13.795 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a54810 2024-03-11T21:24:13.795 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a53d80 2024-03-11T21:24:13.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a53eb0 2024-03-11T21:24:13.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a54000 2024-03-11T21:24:13.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a54960 2024-03-11T21:24:13.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a54360 2024-03-11T21:24:13.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a54490 2024-03-11T21:24:13.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a545a0 2024-03-11T21:24:13.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a546f0 2024-03-11T21:24:13.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a54f40 2024-03-11T21:24:13.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a54a90 2024-03-11T21:24:13.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a54bc0 2024-03-11T21:24:13.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a54d10 2024-03-11T21:24:13.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a55090 2024-03-11T21:24:13.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a55540 2024-03-11T21:24:13.796 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a55650 2024-03-11T21:24:13.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a55c50 2024-03-11T21:24:13.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a551a0 2024-03-11T21:24:13.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a552d0 2024-03-11T21:24:13.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a55400 2024-03-11T21:24:13.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a55d60 2024-03-11T21:24:13.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a557a0 2024-03-11T21:24:13.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a558f0 2024-03-11T21:24:13.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a55a00 2024-03-11T21:24:13.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a55eb0 2024-03-11T21:24:13.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a56360 2024-03-11T21:24:13.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a564b0 2024-03-11T21:24:13.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a56a90 2024-03-11T21:24:13.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a55fc0 2024-03-11T21:24:13.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a560f0 2024-03-11T21:24:13.797 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a56220 2024-03-11T21:24:13.798 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a56be0 2024-03-11T21:24:13.798 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a565e0 2024-03-11T21:24:13.798 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a56730 2024-03-11T21:24:13.798 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a56840 2024-03-11T21:24:13.798 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a56950 2024-03-11T21:24:13.798 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a571c0 2024-03-11T21:24:13.798 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: waiting 0x558254a56d10 2024-03-11T21:24:13.798 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: done. 2024-03-11T21:24:13.798 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (1810926 ms) 2024-03-11T21:24:13.798 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (1810926 ms total) 2024-03-11T21:24:13.798 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2024-03-11T21:24:13.798 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-03-11T21:24:13.798 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-03-11T21:24:13.798 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-03-11T21:24:13.799 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-03-11T21:24:13.799 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.Exec (19 ms) 2024-03-11T21:24:13.799 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-03-11T21:24:13.799 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (4 ms) 2024-03-11T21:24:13.799 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-03-11T21:24:13.799 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (2 ms) 2024-03-11T21:24:13.799 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-03-11T21:24:14.124 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:13 smithi157 ceph-mon[33359]: osdmap e762: 8 total, 8 up, 8 in 2024-03-11T21:24:14.125 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:13 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:24:14.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:13 smithi129 ceph-mon[30184]: osdmap e762: 8 total, 8 up, 8 in 2024-03-11T21:24:14.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:13 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:24:14.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:13 smithi129 ceph-mon[42066]: osdmap e762: 8 total, 8 up, 8 in 2024-03-11T21:24:14.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:13 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key1"}]: dispatch 2024-03-11T21:24:15.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:14 smithi129 ceph-mon[30184]: pgmap v1659: 196 pgs: 196 active+clean; 583 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:24:15.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:14 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key1"}]': finished 2024-03-11T21:24:15.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:14 smithi129 ceph-mon[30184]: osdmap e763: 8 total, 8 up, 8 in 2024-03-11T21:24:15.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:14 smithi129 ceph-mon[42066]: pgmap v1659: 196 pgs: 196 active+clean; 583 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:24:15.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:14 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key1"}]': finished 2024-03-11T21:24:15.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:14 smithi129 ceph-mon[42066]: osdmap e763: 8 total, 8 up, 8 in 2024-03-11T21:24:15.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:24:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:24:15.224 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:14 smithi157 ceph-mon[33359]: pgmap v1659: 196 pgs: 196 active+clean; 583 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-11T21:24:15.224 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:14 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4113756565' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi129-82761-1","app":"app1","key":"key1"}]': finished 2024-03-11T21:24:15.224 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:14 smithi157 ceph-mon[33359]: osdmap e763: 8 total, 8 up, 8 in 2024-03-11T21:24:16.092 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:16.092 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:16.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:16.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:24:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:24:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:24:17.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:16 smithi129 ceph-mon[30184]: pgmap v1661: 196 pgs: 196 active+clean; 583 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-11T21:24:17.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:16 smithi129 ceph-mon[42066]: pgmap v1661: 196 pgs: 196 active+clean; 583 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-11T21:24:17.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:16 smithi157 ceph-mon[33359]: pgmap v1661: 196 pgs: 196 active+clean; 583 KiB data, 892 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-11T21:24:19.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:18 smithi129 ceph-mon[30184]: pgmap v1662: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 853 B/s wr, 2 op/s 2024-03-11T21:24:19.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:18 smithi129 ceph-mon[42066]: pgmap v1662: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 853 B/s wr, 2 op/s 2024-03-11T21:24:19.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:18 smithi157 ceph-mon[33359]: pgmap v1662: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 853 B/s wr, 2 op/s 2024-03-11T21:24:21.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:20 smithi129 ceph-mon[30184]: pgmap v1663: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 671 B/s rd, 0 op/s 2024-03-11T21:24:21.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:21.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:21.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:20 smithi129 ceph-mon[42066]: pgmap v1663: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 671 B/s rd, 0 op/s 2024-03-11T21:24:21.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:21.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:21.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:20 smithi157 ceph-mon[33359]: pgmap v1663: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 671 B/s rd, 0 op/s 2024-03-11T21:24:21.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:21.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:22.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:22 smithi157 ceph-mon[33359]: pgmap v1664: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-03-11T21:24:23.168 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:22 smithi129 ceph-mon[30184]: pgmap v1664: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-03-11T21:24:23.168 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:22 smithi129 ceph-mon[42066]: pgmap v1664: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-03-11T21:24:25.086 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:24 smithi157 ceph-mon[33359]: pgmap v1665: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:25.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:24:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:24:25.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:24 smithi129 ceph-mon[30184]: pgmap v1665: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:25.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:24 smithi129 ceph-mon[42066]: pgmap v1665: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:25.823 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:25.823 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:25 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:25.823 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:25 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:25.823 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:25.823 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:25 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:25.823 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:25 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:26.094 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:24:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:24:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:24:26.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:26.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:25 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:26.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:25 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:26.825 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:26 smithi129 ceph-mon[42066]: pgmap v1666: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 882 B/s rd, 0 op/s 2024-03-11T21:24:26.825 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:26 smithi129 ceph-mon[30184]: pgmap v1666: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 882 B/s rd, 0 op/s 2024-03-11T21:24:27.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:26 smithi157 ceph-mon[33359]: pgmap v1666: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 882 B/s rd, 0 op/s 2024-03-11T21:24:28.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:28 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:28.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:28 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:28.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:28 smithi129 ceph-mon[30184]: pgmap v1667: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:28.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:28 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:28.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:28 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:28.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:28 smithi129 ceph-mon[42066]: pgmap v1667: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:28.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:28 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:28.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:28 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:28.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:28 smithi157 ceph-mon[33359]: pgmap v1667: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:30.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:30 smithi129 ceph-mon[30184]: pgmap v1668: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:30.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:30 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:30.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:30 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:30.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:30 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:30.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:30 smithi129 ceph-mon[42066]: pgmap v1668: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:30.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:30 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:30.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:30 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:30.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:30 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:30.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:30 smithi157 ceph-mon[33359]: pgmap v1668: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:30.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:30 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:30.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:30 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:30.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:30 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:32.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5005 ms) 2024-03-11T21:24:32.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-03-11T21:24:32.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-03-11T21:24:32.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-03-11T21:24:32.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-03-11T21:24:32.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-03-11T21:24:32.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (18459 ms) 2024-03-11T21:24:32.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] 8 tests from LibRadosMisc (23491 ms total) 2024-03-11T21:24:32.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: 2024-03-11T21:24:32.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [----------] Global test environment tear-down 2024-03-11T21:24:32.619 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (1838874 ms total) 2024-03-11T21:24:32.620 INFO:tasks.workunit.client.0.smithi129.stdout: api_misc: [ PASSED ] 12 tests. 2024-03-11T21:24:32.624 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.624 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82954 2024-03-11T21:24:32.624 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82954 2024-03-11T21:24:32.624 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.624 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83376 2024-03-11T21:24:32.625 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83376 2024-03-11T21:24:32.625 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.625 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83410 2024-03-11T21:24:32.625 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83410 2024-03-11T21:24:32.625 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.625 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82617 2024-03-11T21:24:32.625 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82617 2024-03-11T21:24:32.625 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.625 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83255 2024-03-11T21:24:32.625 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83255 2024-03-11T21:24:32.625 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.625 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83173 2024-03-11T21:24:32.625 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83173 2024-03-11T21:24:32.625 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82497 2024-03-11T21:24:32.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82497 2024-03-11T21:24:32.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82560 2024-03-11T21:24:32.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82560 2024-03-11T21:24:32.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83188 2024-03-11T21:24:32.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83188 2024-03-11T21:24:32.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82488 2024-03-11T21:24:32.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82488 2024-03-11T21:24:32.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82508 2024-03-11T21:24:32.626 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82508 2024-03-11T21:24:32.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83555 2024-03-11T21:24:32.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83555 2024-03-11T21:24:32.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82794 2024-03-11T21:24:32.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82794 2024-03-11T21:24:32.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83737 2024-03-11T21:24:32.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83737 2024-03-11T21:24:32.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82925 2024-03-11T21:24:32.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82925 2024-03-11T21:24:32.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83317 2024-03-11T21:24:32.627 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83317 2024-03-11T21:24:32.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83830 2024-03-11T21:24:32.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83830 2024-03-11T21:24:32.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82816 2024-03-11T21:24:32.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82816 2024-03-11T21:24:32.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83208 2024-03-11T21:24:32.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83208 2024-03-11T21:24:32.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83454 2024-03-11T21:24:32.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83454 2024-03-11T21:24:32.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83290 2024-03-11T21:24:32.628 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83290 2024-03-11T21:24:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83651 2024-03-11T21:24:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83651 2024-03-11T21:24:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83115 2024-03-11T21:24:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83115 2024-03-11T21:24:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82651 2024-03-11T21:24:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82651 2024-03-11T21:24:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83752 2024-03-11T21:24:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83752 2024-03-11T21:24:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.629 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83049 2024-03-11T21:24:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83049 2024-03-11T21:24:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82481 2024-03-11T21:24:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82481 2024-03-11T21:24:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82540 2024-03-11T21:24:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82540 2024-03-11T21:24:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82970 2024-03-11T21:24:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82970 2024-03-11T21:24:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=82576 2024-03-11T21:24:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 82576 2024-03-11T21:24:32.630 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83062 2024-03-11T21:24:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83062 2024-03-11T21:24:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83100 2024-03-11T21:24:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83100 2024-03-11T21:24:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83441 2024-03-11T21:24:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83441 2024-03-11T21:24:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ for t in "${!pids[@]}" 2024-03-11T21:24:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=83846 2024-03-11T21:24:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 83846 2024-03-11T21:24:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ exit 0 2024-03-11T21:24:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ cleanup 2024-03-11T21:24:32.631 INFO:tasks.workunit.client.0.smithi129.stderr:+ pkill -P 82475 2024-03-11T21:24:32.722 INFO:tasks.workunit.client.0.smithi129.stderr:+ true 2024-03-11T21:24:32.722 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-11T21:24:32.722 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-11T21:24:32.753 INFO:tasks.workunit:Running workunits matching rados/test_pool_quota.sh on client.0... 2024-03-11T21:24:32.755 INFO:tasks.workunit:Running workunit rados/test_pool_quota.sh... 2024-03-11T21:24:32.755 DEBUG:teuthology.orchestra.run.smithi129:workunit test rados/test_pool_quota.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_quota.sh 2024-03-11T21:24:32.813 INFO:tasks.workunit.client.0.smithi129.stderr:++ uuidgen 2024-03-11T21:24:32.821 INFO:tasks.workunit.client.0.smithi129.stderr:+ p=ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 2024-03-11T21:24:32.821 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph osd pool create ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 12 2024-03-11T21:24:32.905 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.918+0000 7f385fe8f640 1 -- 172.21.15.129:0/1746706131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38581054f0 msgr2=0x7f385810d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:32.905 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.918+0000 7f385fe8f640 1 --2- 172.21.15.129:0/1746706131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38581054f0 0x7f385810d0c0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7f3850009be0 tx=0x7f385002bef0 comp rx=0 tx=0).stop 2024-03-11T21:24:32.905 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.918+0000 7f385fe8f640 1 -- 172.21.15.129:0/1746706131 shutdown_connections 2024-03-11T21:24:32.905 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.918+0000 7f385fe8f640 1 --2- 172.21.15.129:0/1746706131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38581054f0 0x7f385810d0c0 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:32.905 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385fe8f640 1 --2- 172.21.15.129:0/1746706131 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3858100dd0 0x7f3858104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:32.905 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385fe8f640 1 --2- 172.21.15.129:0/1746706131 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3858100420 0x7f3858100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:32.905 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385fe8f640 1 -- 172.21.15.129:0/1746706131 >> 172.21.15.129:0/1746706131 conn(0x7f3858077310 msgr2=0x7f3858077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:32.905 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385fe8f640 1 -- 172.21.15.129:0/1746706131 shutdown_connections 2024-03-11T21:24:32.905 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385fe8f640 1 -- 172.21.15.129:0/1746706131 wait complete. 2024-03-11T21:24:32.905 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385fe8f640 1 Processor -- start 2024-03-11T21:24:32.905 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385fe8f640 1 -- start start 2024-03-11T21:24:32.906 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385fe8f640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3858100420 0x7f3858197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:32.906 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385fe8f640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38581054f0 0x7f38581984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:32.906 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385fe8f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3858198a30 0x7f38581a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:32.906 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385fe8f640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385810f9d0 con 0x7f38581054f0 2024-03-11T21:24:32.906 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385fe8f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385810f850 con 0x7f3858198a30 2024-03-11T21:24:32.906 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385fe8f640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385810fb50 con 0x7f3858100420 2024-03-11T21:24:32.906 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385dc2b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3858100420 0x7f3858197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:32.906 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385dc2b640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3858100420 0x7f3858197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:38832/0 (socket says 172.21.15.129:38832) 2024-03-11T21:24:32.906 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385d42a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38581054f0 0x7f38581984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:32.906 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.919+0000 7f385dc2b640 1 -- 172.21.15.129:0/3398685733 learned_addr learned my addr 172.21.15.129:0/3398685733 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T21:24:32.906 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.920+0000 7f385e42c640 1 --2- 172.21.15.129:0/3398685733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3858198a30 0x7f38581a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:32.906 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.920+0000 7f385dc2b640 1 -- 172.21.15.129:0/3398685733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3858198a30 msgr2=0x7f38581a2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:32.906 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.920+0000 7f385dc2b640 1 --2- 172.21.15.129:0/3398685733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3858198a30 0x7f38581a2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:32.907 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.920+0000 7f385dc2b640 1 -- 172.21.15.129:0/3398685733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38581054f0 msgr2=0x7f38581984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:32.907 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.920+0000 7f385dc2b640 1 --2- 172.21.15.129:0/3398685733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38581054f0 0x7f38581984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:32.907 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.920+0000 7f385dc2b640 1 -- 172.21.15.129:0/3398685733 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38581a37e0 con 0x7f3858100420 2024-03-11T21:24:32.907 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.920+0000 7f385e42c640 1 --2- 172.21.15.129:0/3398685733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3858198a30 0x7f38581a2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T21:24:32.907 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.920+0000 7f385d42a640 1 --2- 172.21.15.129:0/3398685733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38581054f0 0x7f38581984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:24:32.907 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.920+0000 7f385dc2b640 1 --2- 172.21.15.129:0/3398685733 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3858100420 0x7f3858197f60 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7f385400e9c0 tx=0x7f385400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:32.907 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.920+0000 7f384effd640 1 -- 172.21.15.129:0/3398685733 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385400cde0 con 0x7f3858100420 2024-03-11T21:24:32.907 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.920+0000 7f384effd640 1 -- 172.21.15.129:0/3398685733 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f38540035f0 con 0x7f3858100420 2024-03-11T21:24:32.907 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.920+0000 7f385fe8f640 1 -- 172.21.15.129:0/3398685733 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38581a3ad0 con 0x7f3858100420 2024-03-11T21:24:32.907 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.920+0000 7f384effd640 1 -- 172.21.15.129:0/3398685733 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38540044a0 con 0x7f3858100420 2024-03-11T21:24:32.907 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.920+0000 7f385fe8f640 1 -- 172.21.15.129:0/3398685733 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38581a4020 con 0x7f3858100420 2024-03-11T21:24:32.907 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.921+0000 7f385fe8f640 1 -- 172.21.15.129:0/3398685733 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38580681d0 con 0x7f3858100420 2024-03-11T21:24:32.907 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.921+0000 7f384effd640 1 -- 172.21.15.129:0/3398685733 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f3854004000 con 0x7f3858100420 2024-03-11T21:24:32.908 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.922+0000 7f384effd640 1 --2- 172.21.15.129:0/3398685733 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f384406ddf0 0x7f38440702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:32.908 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.922+0000 7f384effd640 1 -- 172.21.15.129:0/3398685733 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(764..764 src has 254..764) v4 ==== 8631+0+0 (secure 0 0 0) 0x7f385408eb20 con 0x7f3858100420 2024-03-11T21:24:32.908 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.922+0000 7f385d42a640 1 --2- 172.21.15.129:0/3398685733 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f384406ddf0 0x7f38440702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:32.908 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.922+0000 7f385d42a640 1 --2- 172.21.15.129:0/3398685733 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f384406ddf0 0x7f38440702b0 secure :-1 s=READY pgs=4022 cs=0 l=1 rev1=1 crypto rx=0x7f3858069190 tx=0x7f3848005f90 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:32.910 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:32.927+0000 7f384effd640 1 -- 172.21.15.129:0/3398685733 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f385405bb50 con 0x7f3858100420 2024-03-11T21:24:32.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:32 smithi129 ceph-mon[30184]: pgmap v1669: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:32.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:32 smithi129 ceph-mon[42066]: pgmap v1669: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:32.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:32 smithi157 ceph-mon[33359]: pgmap v1669: 196 pgs: 196 active+clean; 583 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:24:33.086 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.102+0000 7f385fe8f640 1 -- 172.21.15.129:0/3398685733 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12} v 0) v1 -- 0x7f3858100dd0 con 0x7f3858100420 2024-03-11T21:24:33.617 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.633+0000 7f384effd640 1 -- 172.21.15.129:0/3398685733 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]=0 pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' created v765) v1 ==== 176+0+0 (secure 0 0 0) 0x7f385405f5a0 con 0x7f3858100420 2024-03-11T21:24:33.722 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.739+0000 7f385fe8f640 1 -- 172.21.15.129:0/3398685733 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12} v 0) v1 -- 0x7f3858100fa0 con 0x7f3858100420 2024-03-11T21:24:33.723 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.739+0000 7f384effd640 1 -- 172.21.15.129:0/3398685733 <== mon.2 v2:172.21.15.129:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]=0 pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' already exists v765) v1 ==== 183+0+0 (secure 0 0 0) 0x7f3854063b60 con 0x7f3858100420 2024-03-11T21:24:33.723 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' already exists 2024-03-11T21:24:33.725 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.741+0000 7f385fe8f640 1 -- 172.21.15.129:0/3398685733 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f384406ddf0 msgr2=0x7f38440702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:33.725 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.741+0000 7f385fe8f640 1 --2- 172.21.15.129:0/3398685733 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f384406ddf0 0x7f38440702b0 secure :-1 s=READY pgs=4022 cs=0 l=1 rev1=1 crypto rx=0x7f3858069190 tx=0x7f3848005f90 comp rx=0 tx=0).stop 2024-03-11T21:24:33.725 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.741+0000 7f385fe8f640 1 -- 172.21.15.129:0/3398685733 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3858100420 msgr2=0x7f3858197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:33.725 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.741+0000 7f385fe8f640 1 --2- 172.21.15.129:0/3398685733 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3858100420 0x7f3858197f60 secure :-1 s=READY pgs=3060 cs=0 l=1 rev1=1 crypto rx=0x7f385400e9c0 tx=0x7f385400ee80 comp rx=0 tx=0).stop 2024-03-11T21:24:33.725 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.741+0000 7f385fe8f640 1 -- 172.21.15.129:0/3398685733 shutdown_connections 2024-03-11T21:24:33.725 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.741+0000 7f385fe8f640 1 --2- 172.21.15.129:0/3398685733 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f384406ddf0 0x7f38440702b0 unknown :-1 s=CLOSED pgs=4022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:33.726 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.741+0000 7f385fe8f640 1 --2- 172.21.15.129:0/3398685733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3858198a30 0x7f38581a2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:33.726 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.741+0000 7f385fe8f640 1 --2- 172.21.15.129:0/3398685733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38581054f0 0x7f38581984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:33.726 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.741+0000 7f385fe8f640 1 --2- 172.21.15.129:0/3398685733 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3858100420 0x7f3858197f60 unknown :-1 s=CLOSED pgs=3060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:33.726 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.741+0000 7f385fe8f640 1 -- 172.21.15.129:0/3398685733 >> 172.21.15.129:0/3398685733 conn(0x7f3858077310 msgr2=0x7f38580fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:33.726 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.741+0000 7f385fe8f640 1 -- 172.21.15.129:0/3398685733 shutdown_connections 2024-03-11T21:24:33.726 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.741+0000 7f385fe8f640 1 -- 172.21.15.129:0/3398685733 wait complete. 2024-03-11T21:24:33.735 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph osd pool set-quota ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 max_objects 10 2024-03-11T21:24:33.820 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.834+0000 7f445d6c0640 1 -- 172.21.15.129:0/51364759 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f44581054f0 msgr2=0x7f445810d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:33.820 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.834+0000 7f445d6c0640 1 --2- 172.21.15.129:0/51364759 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f44581054f0 0x7f445810d0c0 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7f4450009ca0 tx=0x7f445002c0f0 comp rx=0 tx=0).stop 2024-03-11T21:24:33.820 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.834+0000 7f445d6c0640 1 -- 172.21.15.129:0/51364759 shutdown_connections 2024-03-11T21:24:33.820 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.834+0000 7f445d6c0640 1 --2- 172.21.15.129:0/51364759 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f44581054f0 0x7f445810d0c0 unknown :-1 s=CLOSED pgs=3061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:33.820 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.834+0000 7f445d6c0640 1 --2- 172.21.15.129:0/51364759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4458100dd0 0x7f4458104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:33.820 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.834+0000 7f445d6c0640 1 --2- 172.21.15.129:0/51364759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4458100420 0x7f4458100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:33.821 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.834+0000 7f445d6c0640 1 -- 172.21.15.129:0/51364759 >> 172.21.15.129:0/51364759 conn(0x7f4458077400 msgr2=0x7f4458077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:33.821 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.834+0000 7f445d6c0640 1 -- 172.21.15.129:0/51364759 shutdown_connections 2024-03-11T21:24:33.821 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.834+0000 7f445d6c0640 1 -- 172.21.15.129:0/51364759 wait complete. 2024-03-11T21:24:33.821 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f445d6c0640 1 Processor -- start 2024-03-11T21:24:33.821 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f445d6c0640 1 -- start start 2024-03-11T21:24:33.821 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f445d6c0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4458100420 0x7f4458101eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:33.821 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f445d6c0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44581054f0 0x7f44581023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:33.821 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f445d6c0640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4458102930 0x7f44581a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:33.821 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f445d6c0640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445810fa10 con 0x7f44581054f0 2024-03-11T21:24:33.821 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f445d6c0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445810f890 con 0x7f4458100420 2024-03-11T21:24:33.821 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f445d6c0640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445810fb90 con 0x7f4458102930 2024-03-11T21:24:33.821 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f444ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44581054f0 0x7f44581023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:33.822 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f44577fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4458102930 0x7f44581a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:33.822 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f444ffff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44581054f0 0x7f44581023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47626/0 (socket says 172.21.15.129:47626) 2024-03-11T21:24:33.822 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f444ffff640 1 -- 172.21.15.129:0/3640394356 learned_addr learned my addr 172.21.15.129:0/3640394356 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T21:24:33.822 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f4456ffd640 1 --2- 172.21.15.129:0/3640394356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4458100420 0x7f4458101eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:33.822 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f44577fe640 1 -- 172.21.15.129:0/3640394356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4458100420 msgr2=0x7f4458101eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:33.822 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f44577fe640 1 --2- 172.21.15.129:0/3640394356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4458100420 0x7f4458101eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:33.822 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f44577fe640 1 -- 172.21.15.129:0/3640394356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44581054f0 msgr2=0x7f44581023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:33.822 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f44577fe640 1 --2- 172.21.15.129:0/3640394356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44581054f0 0x7f44581023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:33.822 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f44577fe640 1 -- 172.21.15.129:0/3640394356 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44581a59b0 con 0x7f4458102930 2024-03-11T21:24:33.822 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f444ffff640 1 --2- 172.21.15.129:0/3640394356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44581054f0 0x7f44581023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:24:33.822 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.835+0000 7f44577fe640 1 --2- 172.21.15.129:0/3640394356 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4458102930 0x7f44581a5310 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f445002c700 tx=0x7f4450032830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:33.822 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.836+0000 7f4454ff9640 1 -- 172.21.15.129:0/3640394356 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450041020 con 0x7f4458102930 2024-03-11T21:24:33.822 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.836+0000 7f445d6c0640 1 -- 172.21.15.129:0/3640394356 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44581a5c40 con 0x7f4458102930 2024-03-11T21:24:33.823 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.836+0000 7f4454ff9640 1 -- 172.21.15.129:0/3640394356 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f4450003df0 con 0x7f4458102930 2024-03-11T21:24:33.823 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.836+0000 7f4454ff9640 1 -- 172.21.15.129:0/3640394356 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450039040 con 0x7f4458102930 2024-03-11T21:24:33.823 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.836+0000 7f445d6c0640 1 -- 172.21.15.129:0/3640394356 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44581a6100 con 0x7f4458102930 2024-03-11T21:24:33.823 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.837+0000 7f445d6c0640 1 -- 172.21.15.129:0/3640394356 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4458068620 con 0x7f4458102930 2024-03-11T21:24:33.824 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.837+0000 7f4454ff9640 1 -- 172.21.15.129:0/3640394356 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f4450044020 con 0x7f4458102930 2024-03-11T21:24:33.824 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.837+0000 7f4454ff9640 1 --2- 172.21.15.129:0/3640394356 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f444406ddf0 0x7f44440702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:33.824 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.838+0000 7f4454ff9640 1 -- 172.21.15.129:0/3640394356 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(765..765 src has 254..765) v4 ==== 9006+0+0 (secure 0 0 0) 0x7f44500b0930 con 0x7f4458102930 2024-03-11T21:24:33.824 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.838+0000 7f4456ffd640 1 --2- 172.21.15.129:0/3640394356 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f444406ddf0 0x7f44440702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:33.824 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.838+0000 7f4456ffd640 1 --2- 172.21.15.129:0/3640394356 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f444406ddf0 0x7f44440702b0 secure :-1 s=READY pgs=4023 cs=0 l=1 rev1=1 crypto rx=0x7f44480058c0 tx=0x7f4448005c60 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:33.826 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:33.842+0000 7f4454ff9640 1 -- 172.21.15.129:0/3640394356 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f445004b020 con 0x7f4458102930 2024-03-11T21:24:33.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:33 smithi129 ceph-mon[30184]: osdmap e764: 8 total, 8 up, 8 in 2024-03-11T21:24:33.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:33 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3398685733' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]: dispatch 2024-03-11T21:24:33.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:33 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]: dispatch 2024-03-11T21:24:33.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:33 smithi129 ceph-mon[42066]: osdmap e764: 8 total, 8 up, 8 in 2024-03-11T21:24:33.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:33 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3398685733' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]: dispatch 2024-03-11T21:24:33.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:33 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]: dispatch 2024-03-11T21:24:33.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:33 smithi157 ceph-mon[33359]: osdmap e764: 8 total, 8 up, 8 in 2024-03-11T21:24:33.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:33 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3398685733' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]: dispatch 2024-03-11T21:24:33.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:33 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]: dispatch 2024-03-11T21:24:34.004 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:34.020+0000 7f445d6c0640 1 -- 172.21.15.129:0/3640394356 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f4458100dd0 con 0x7f4458102930 2024-03-11T21:24:34.614 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:34.630+0000 7f4454ff9640 1 -- 172.21.15.129:0/3640394356 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 v766) v1 ==== 223+0+0 (secure 0 0 0) 0x7f445007d7f0 con 0x7f4458102930 2024-03-11T21:24:34.724 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:34.741+0000 7f445d6c0640 1 -- 172.21.15.129:0/3640394356 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f44581a6330 con 0x7f4458102930 2024-03-11T21:24:34.890 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:34 smithi157 ceph-mon[33359]: pgmap v1671: 164 pgs: 164 active+clean; 582 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-11T21:24:34.890 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:34 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]': finished 2024-03-11T21:24:34.890 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:34 smithi157 ceph-mon[33359]: osdmap e765: 8 total, 8 up, 8 in 2024-03-11T21:24:34.891 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:34 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3398685733' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]: dispatch 2024-03-11T21:24:34.891 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:34 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]: dispatch 2024-03-11T21:24:34.891 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:34 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3640394356' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:24:34.891 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:34 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:24:34.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:34 smithi129 ceph-mon[30184]: pgmap v1671: 164 pgs: 164 active+clean; 582 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-11T21:24:34.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:34 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]': finished 2024-03-11T21:24:34.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:34 smithi129 ceph-mon[30184]: osdmap e765: 8 total, 8 up, 8 in 2024-03-11T21:24:34.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:34 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3398685733' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]: dispatch 2024-03-11T21:24:34.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:34 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]: dispatch 2024-03-11T21:24:34.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:34 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3640394356' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:24:34.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:34 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:24:34.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:34 smithi129 ceph-mon[42066]: pgmap v1671: 164 pgs: 164 active+clean; 582 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-11T21:24:34.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:34 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]': finished 2024-03-11T21:24:34.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:34 smithi129 ceph-mon[42066]: osdmap e765: 8 total, 8 up, 8 in 2024-03-11T21:24:34.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:34 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3398685733' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]: dispatch 2024-03-11T21:24:34.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:34 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pg_num": 12}]: dispatch 2024-03-11T21:24:34.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:34 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3640394356' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:24:34.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:34 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:24:35.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:24:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:24:35.628 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.644+0000 7f4454ff9640 1 -- 172.21.15.129:0/3640394356 <== mon.2 v2:172.21.15.129:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 v767) v1 ==== 223+0+0 (secure 0 0 0) 0x7f4450081240 con 0x7f4458102930 2024-03-11T21:24:35.628 INFO:tasks.workunit.client.0.smithi129.stderr:set-quota max_objects = 10 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 2024-03-11T21:24:35.630 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.646+0000 7f445d6c0640 1 -- 172.21.15.129:0/3640394356 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f444406ddf0 msgr2=0x7f44440702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:35.631 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.647+0000 7f445d6c0640 1 --2- 172.21.15.129:0/3640394356 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f444406ddf0 0x7f44440702b0 secure :-1 s=READY pgs=4023 cs=0 l=1 rev1=1 crypto rx=0x7f44480058c0 tx=0x7f4448005c60 comp rx=0 tx=0).stop 2024-03-11T21:24:35.631 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.647+0000 7f445d6c0640 1 -- 172.21.15.129:0/3640394356 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4458102930 msgr2=0x7f44581a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:35.631 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.647+0000 7f445d6c0640 1 --2- 172.21.15.129:0/3640394356 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4458102930 0x7f44581a5310 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f445002c700 tx=0x7f4450032830 comp rx=0 tx=0).stop 2024-03-11T21:24:35.631 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.647+0000 7f445d6c0640 1 -- 172.21.15.129:0/3640394356 shutdown_connections 2024-03-11T21:24:35.631 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.647+0000 7f445d6c0640 1 --2- 172.21.15.129:0/3640394356 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f444406ddf0 0x7f44440702b0 unknown :-1 s=CLOSED pgs=4023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:35.631 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.647+0000 7f445d6c0640 1 --2- 172.21.15.129:0/3640394356 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f4458102930 0x7f44581a5310 unknown :-1 s=CLOSED pgs=3062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:35.631 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.647+0000 7f445d6c0640 1 --2- 172.21.15.129:0/3640394356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44581054f0 0x7f44581023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:35.631 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.647+0000 7f445d6c0640 1 --2- 172.21.15.129:0/3640394356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4458100420 0x7f4458101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:35.631 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.647+0000 7f445d6c0640 1 -- 172.21.15.129:0/3640394356 >> 172.21.15.129:0/3640394356 conn(0x7f4458077400 msgr2=0x7f4458103ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:35.631 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.647+0000 7f445d6c0640 1 -- 172.21.15.129:0/3640394356 shutdown_connections 2024-03-11T21:24:35.631 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.647+0000 7f445d6c0640 1 -- 172.21.15.129:0/3640394356 wait complete. 2024-03-11T21:24:35.642 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph osd pool application enable ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 rados 2024-03-11T21:24:35.732 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.746+0000 7f720bcdf640 1 -- 172.21.15.129:0/2084863112 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7204101530 msgr2=0x7f7204101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:35.732 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.746+0000 7f720bcdf640 1 --2- 172.21.15.129:0/2084863112 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7204101530 0x7f7204101910 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7f71f4009860 tx=0x7f71f402bb70 comp rx=0 tx=0).stop 2024-03-11T21:24:35.732 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.746+0000 7f720bcdf640 1 -- 172.21.15.129:0/2084863112 shutdown_connections 2024-03-11T21:24:35.732 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.746+0000 7f720bcdf640 1 --2- 172.21.15.129:0/2084863112 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7204106600 0x7f720410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:35.732 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.746+0000 7f720bcdf640 1 --2- 172.21.15.129:0/2084863112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7204101ee0 0x7f7204105f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:35.732 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.746+0000 7f720bcdf640 1 --2- 172.21.15.129:0/2084863112 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f7204101530 0x7f7204101910 unknown :-1 s=CLOSED pgs=3063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:35.732 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.746+0000 7f720bcdf640 1 -- 172.21.15.129:0/2084863112 >> 172.21.15.129:0/2084863112 conn(0x7f72040fcdf0 msgr2=0x7f72040ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:35.732 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.746+0000 7f720bcdf640 1 -- 172.21.15.129:0/2084863112 shutdown_connections 2024-03-11T21:24:35.733 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.746+0000 7f720bcdf640 1 -- 172.21.15.129:0/2084863112 wait complete. 2024-03-11T21:24:35.733 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.746+0000 7f720bcdf640 1 Processor -- start 2024-03-11T21:24:35.733 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.746+0000 7f720bcdf640 1 -- start start 2024-03-11T21:24:35.733 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720bcdf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7204101530 0x7f7204197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:35.733 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720bcdf640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7204106600 0x7f7204197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:35.733 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720bcdf640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f720419b9e0 0x7f720419be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:35.733 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720bcdf640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7204110b30 con 0x7f7204106600 2024-03-11T21:24:35.733 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720bcdf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72041109b0 con 0x7f7204101530 2024-03-11T21:24:35.733 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720bcdf640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7204110cb0 con 0x7f720419b9e0 2024-03-11T21:24:35.733 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720a27c640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f720419b9e0 0x7f720419be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:35.733 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720a27c640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f720419b9e0 0x7f720419be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55446/0 (socket says 172.21.15.129:55446) 2024-03-11T21:24:35.734 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720927a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7204106600 0x7f7204197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:35.734 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720a27c640 1 -- 172.21.15.129:0/303267244 learned_addr learned my addr 172.21.15.129:0/303267244 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T21:24:35.734 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f7209a7b640 1 --2- 172.21.15.129:0/303267244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7204101530 0x7f7204197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:35.734 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720a27c640 1 -- 172.21.15.129:0/303267244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7204101530 msgr2=0x7f7204197120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:35.734 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720a27c640 1 --2- 172.21.15.129:0/303267244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7204101530 0x7f7204197120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:35.734 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720a27c640 1 -- 172.21.15.129:0/303267244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7204106600 msgr2=0x7f7204197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:35.734 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720a27c640 1 --2- 172.21.15.129:0/303267244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7204106600 0x7f7204197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:35.734 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720a27c640 1 -- 172.21.15.129:0/303267244 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720419c4d0 con 0x7f720419b9e0 2024-03-11T21:24:35.734 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720927a640 1 --2- 172.21.15.129:0/303267244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7204106600 0x7f7204197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:24:35.734 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720a27c640 1 --2- 172.21.15.129:0/303267244 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f720419b9e0 0x7f720419be90 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00efc0 tx=0x7f71fc00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:35.734 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f71faffd640 1 -- 172.21.15.129:0/303267244 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc003970 con 0x7f720419b9e0 2024-03-11T21:24:35.735 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720bcdf640 1 -- 172.21.15.129:0/303267244 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f720419c7c0 con 0x7f720419b9e0 2024-03-11T21:24:35.735 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f71faffd640 1 -- 172.21.15.129:0/303267244 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f71fc004020 con 0x7f720419b9e0 2024-03-11T21:24:35.735 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f71faffd640 1 -- 172.21.15.129:0/303267244 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc00b870 con 0x7f720419b9e0 2024-03-11T21:24:35.735 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.747+0000 7f720bcdf640 1 -- 172.21.15.129:0/303267244 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72041a3fb0 con 0x7f720419b9e0 2024-03-11T21:24:35.735 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.748+0000 7f720bcdf640 1 -- 172.21.15.129:0/303267244 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7204068620 con 0x7f720419b9e0 2024-03-11T21:24:35.736 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.749+0000 7f71faffd640 1 -- 172.21.15.129:0/303267244 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f71fc01d020 con 0x7f720419b9e0 2024-03-11T21:24:35.736 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.749+0000 7f71faffd640 1 --2- 172.21.15.129:0/303267244 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f71f006ddf0 0x7f71f00702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:24:35.736 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.749+0000 7f71faffd640 1 -- 172.21.15.129:0/303267244 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(767..767 src has 254..767) v4 ==== 9006+0+0 (secure 0 0 0) 0x7f71fc08e650 con 0x7f720419b9e0 2024-03-11T21:24:35.736 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.749+0000 7f7209a7b640 1 --2- 172.21.15.129:0/303267244 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f71f006ddf0 0x7f71f00702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:24:35.736 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.750+0000 7f7209a7b640 1 --2- 172.21.15.129:0/303267244 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f71f006ddf0 0x7f71f00702b0 secure :-1 s=READY pgs=4024 cs=0 l=1 rev1=1 crypto rx=0x7f71f4009830 tx=0x7f71f402e040 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:24:35.737 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.754+0000 7f71faffd640 1 -- 172.21.15.129:0/303267244 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f71fc05ef60 con 0x7f720419b9e0 2024-03-11T21:24:35.918 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:35.934+0000 7f720bcdf640 1 -- 172.21.15.129:0/303267244 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"} v 0) v1 -- 0x7f7204101ee0 con 0x7f720419b9e0 2024-03-11T21:24:35.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:35 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]': finished 2024-03-11T21:24:35.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:35 smithi129 ceph-mon[30184]: osdmap e766: 8 total, 8 up, 8 in 2024-03-11T21:24:35.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:35 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3640394356' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:24:35.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:35 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:24:35.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:35 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:35.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:35.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:35 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]': finished 2024-03-11T21:24:35.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:35 smithi129 ceph-mon[42066]: osdmap e766: 8 total, 8 up, 8 in 2024-03-11T21:24:35.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:35 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3640394356' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:24:35.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:35 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:24:35.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:35 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:35.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:35.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:35 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]': finished 2024-03-11T21:24:35.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:35 smithi157 ceph-mon[33359]: osdmap e766: 8 total, 8 up, 8 in 2024-03-11T21:24:35.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:35 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3640394356' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:24:35.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:35 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:24:35.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:35 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:35.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:36.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:24:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:24:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:24:36.634 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:36.650+0000 7f71faffd640 1 -- 172.21.15.129:0/303267244 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]=0 enabled application 'rados' on pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' v768) v1 ==== 213+0+0 (secure 0 0 0) 0x7f71fc063520 con 0x7f720419b9e0 2024-03-11T21:24:36.742 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:36.758+0000 7f720bcdf640 1 -- 172.21.15.129:0/303267244 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"} v 0) v1 -- 0x7f72041a4240 con 0x7f720419b9e0 2024-03-11T21:24:36.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:36 smithi129 ceph-mon[42066]: pgmap v1674: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:24:36.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:36 smithi129 ceph-mon[42066]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:24:36.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]': finished 2024-03-11T21:24:36.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:36 smithi129 ceph-mon[42066]: osdmap e767: 8 total, 8 up, 8 in 2024-03-11T21:24:36.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:36 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/303267244' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]: dispatch 2024-03-11T21:24:36.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:36 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]: dispatch 2024-03-11T21:24:36.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:36 smithi129 ceph-mon[30184]: pgmap v1674: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:24:36.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:36 smithi129 ceph-mon[30184]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:24:36.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]': finished 2024-03-11T21:24:36.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:36 smithi129 ceph-mon[30184]: osdmap e767: 8 total, 8 up, 8 in 2024-03-11T21:24:36.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:36 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/303267244' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]: dispatch 2024-03-11T21:24:36.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:36 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]: dispatch 2024-03-11T21:24:36.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:36 smithi157 ceph-mon[33359]: pgmap v1674: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:24:36.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:36 smithi157 ceph-mon[33359]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:24:36.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "10"}]': finished 2024-03-11T21:24:36.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:36 smithi157 ceph-mon[33359]: osdmap e767: 8 total, 8 up, 8 in 2024-03-11T21:24:36.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:36 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/303267244' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]: dispatch 2024-03-11T21:24:36.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:36 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]: dispatch 2024-03-11T21:24:37.643 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:37.659+0000 7f71faffd640 1 -- 172.21.15.129:0/303267244 <== mon.2 v2:172.21.15.129:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]=0 enabled application 'rados' on pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' v769) v1 ==== 213+0+0 (secure 0 0 0) 0x7f71fc058f60 con 0x7f720419b9e0 2024-03-11T21:24:37.643 INFO:tasks.workunit.client.0.smithi129.stderr:enabled application 'rados' on pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' 2024-03-11T21:24:37.645 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:37.662+0000 7f720bcdf640 1 -- 172.21.15.129:0/303267244 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f71f006ddf0 msgr2=0x7f71f00702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:37.646 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:37.662+0000 7f720bcdf640 1 --2- 172.21.15.129:0/303267244 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f71f006ddf0 0x7f71f00702b0 secure :-1 s=READY pgs=4024 cs=0 l=1 rev1=1 crypto rx=0x7f71f4009830 tx=0x7f71f402e040 comp rx=0 tx=0).stop 2024-03-11T21:24:37.646 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:37.662+0000 7f720bcdf640 1 -- 172.21.15.129:0/303267244 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f720419b9e0 msgr2=0x7f720419be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:24:37.646 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:37.662+0000 7f720bcdf640 1 --2- 172.21.15.129:0/303267244 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f720419b9e0 0x7f720419be90 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00efc0 tx=0x7f71fc00c800 comp rx=0 tx=0).stop 2024-03-11T21:24:37.646 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:37.662+0000 7f720bcdf640 1 -- 172.21.15.129:0/303267244 shutdown_connections 2024-03-11T21:24:37.646 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:37.662+0000 7f720bcdf640 1 --2- 172.21.15.129:0/303267244 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f71f006ddf0 0x7f71f00702b0 unknown :-1 s=CLOSED pgs=4024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:37.646 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:37.662+0000 7f720bcdf640 1 --2- 172.21.15.129:0/303267244 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f720419b9e0 0x7f720419be90 unknown :-1 s=CLOSED pgs=3064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:37.646 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:37.662+0000 7f720bcdf640 1 --2- 172.21.15.129:0/303267244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7204106600 0x7f7204197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:37.646 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:37.662+0000 7f720bcdf640 1 --2- 172.21.15.129:0/303267244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7204101530 0x7f7204197120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:24:37.646 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:37.662+0000 7f720bcdf640 1 -- 172.21.15.129:0/303267244 >> 172.21.15.129:0/303267244 conn(0x7f72040fcdf0 msgr2=0x7f72040fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:24:37.647 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:37.662+0000 7f720bcdf640 1 -- 172.21.15.129:0/303267244 shutdown_connections 2024-03-11T21:24:37.647 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:24:37.662+0000 7f720bcdf640 1 -- 172.21.15.129:0/303267244 wait complete. 2024-03-11T21:24:37.657 INFO:tasks.workunit.client.0.smithi129.stderr:++ seq 1 10 2024-03-11T21:24:37.671 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:24:37.671 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 put obj1 /etc/passwd 2024-03-11T21:24:37.708 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:24:37.708 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 put obj2 /etc/passwd 2024-03-11T21:24:37.744 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:24:37.745 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 put obj3 /etc/passwd 2024-03-11T21:24:37.779 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:24:37.780 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 put obj4 /etc/passwd 2024-03-11T21:24:37.814 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:24:37.814 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 put obj5 /etc/passwd 2024-03-11T21:24:37.852 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:24:37.852 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 put obj6 /etc/passwd 2024-03-11T21:24:37.887 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:24:37.888 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 put obj7 /etc/passwd 2024-03-11T21:24:37.922 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:24:37.922 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 put obj8 /etc/passwd 2024-03-11T21:24:37.956 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:24:37.957 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 put obj9 /etc/passwd 2024-03-11T21:24:37.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]': finished 2024-03-11T21:24:37.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:37 smithi129 ceph-mon[30184]: osdmap e768: 8 total, 8 up, 8 in 2024-03-11T21:24:37.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:37 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/303267244' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]: dispatch 2024-03-11T21:24:37.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:37 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]: dispatch 2024-03-11T21:24:37.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]': finished 2024-03-11T21:24:37.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:37 smithi129 ceph-mon[42066]: osdmap e768: 8 total, 8 up, 8 in 2024-03-11T21:24:37.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:37 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/303267244' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]: dispatch 2024-03-11T21:24:37.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:37 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]: dispatch 2024-03-11T21:24:37.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]': finished 2024-03-11T21:24:37.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:37 smithi157 ceph-mon[33359]: osdmap e768: 8 total, 8 up, 8 in 2024-03-11T21:24:37.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:37 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/303267244' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]: dispatch 2024-03-11T21:24:37.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:37 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]: dispatch 2024-03-11T21:24:37.991 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:24:37.991 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 put obj10 /etc/passwd 2024-03-11T21:24:38.025 INFO:tasks.workunit.client.0.smithi129.stderr:+ sleep 30 2024-03-11T21:24:38.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:38 smithi129 ceph-mon[30184]: pgmap v1677: 176 pgs: 176 active+clean; 582 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:38.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:38 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]': finished 2024-03-11T21:24:38.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:38 smithi129 ceph-mon[30184]: osdmap e769: 8 total, 8 up, 8 in 2024-03-11T21:24:38.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:38 smithi129 ceph-mon[42066]: pgmap v1677: 176 pgs: 176 active+clean; 582 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:38.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:38 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]': finished 2024-03-11T21:24:38.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:38 smithi129 ceph-mon[42066]: osdmap e769: 8 total, 8 up, 8 in 2024-03-11T21:24:38.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:38 smithi157 ceph-mon[33359]: pgmap v1677: 176 pgs: 176 active+clean; 582 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:38.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:38 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "app": "rados"}]': finished 2024-03-11T21:24:38.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:38 smithi157 ceph-mon[33359]: osdmap e769: 8 total, 8 up, 8 in 2024-03-11T21:24:40.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:40 smithi129 ceph-mon[30184]: pgmap v1679: 176 pgs: 176 active+clean; 582 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:24:40.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:40 smithi129 ceph-mon[42066]: pgmap v1679: 176 pgs: 176 active+clean; 582 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:24:40.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:40 smithi157 ceph-mon[33359]: pgmap v1679: 176 pgs: 176 active+clean; 582 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:24:42.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:42 smithi129 ceph-mon[42066]: pgmap v1680: 176 pgs: 176 active+clean; 582 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:24:42.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:42 smithi129 ceph-mon[30184]: pgmap v1680: 176 pgs: 176 active+clean; 582 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:24:42.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:42 smithi157 ceph-mon[33359]: pgmap v1680: 176 pgs: 176 active+clean; 582 KiB data, 880 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:24:44.897 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:44 smithi157 ceph-mon[33359]: pgmap v1681: 176 pgs: 176 active+clean; 697 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-11T21:24:44.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:44 smithi129 ceph-mon[30184]: pgmap v1681: 176 pgs: 176 active+clean; 697 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-11T21:24:44.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:44 smithi129 ceph-mon[42066]: pgmap v1681: 176 pgs: 176 active+clean; 697 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-11T21:24:45.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:24:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:24:45.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:45.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:45 smithi129 ceph-mon[30184]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:24:45.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:45 smithi129 ceph-mon[30184]: pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' is full (reached quota's max_objects: 10) 2024-03-11T21:24:45.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:45 smithi129 ceph-mon[30184]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-11T21:24:45.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:45 smithi129 ceph-mon[30184]: osdmap e770: 8 total, 8 up, 8 in 2024-03-11T21:24:45.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:45 smithi129 ceph-mon[42066]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:24:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:45 smithi129 ceph-mon[42066]: pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' is full (reached quota's max_objects: 10) 2024-03-11T21:24:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:45 smithi129 ceph-mon[42066]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-11T21:24:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:45 smithi129 ceph-mon[42066]: osdmap e770: 8 total, 8 up, 8 in 2024-03-11T21:24:45.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:45.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:45 smithi157 ceph-mon[33359]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:24:45.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:45 smithi157 ceph-mon[33359]: pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' is full (reached quota's max_objects: 10) 2024-03-11T21:24:45.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:45 smithi157 ceph-mon[33359]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-11T21:24:45.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:45 smithi157 ceph-mon[33359]: osdmap e770: 8 total, 8 up, 8 in 2024-03-11T21:24:46.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:24:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:24:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:24:46.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:46 smithi129 ceph-mon[30184]: pgmap v1683: 176 pgs: 176 active+clean; 697 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 15 KiB/s wr, 1 op/s 2024-03-11T21:24:46.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:46 smithi129 ceph-mon[42066]: pgmap v1683: 176 pgs: 176 active+clean; 697 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 15 KiB/s wr, 1 op/s 2024-03-11T21:24:46.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:46 smithi157 ceph-mon[33359]: pgmap v1683: 176 pgs: 176 active+clean; 697 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 15 KiB/s wr, 1 op/s 2024-03-11T21:24:48.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:48 smithi129 ceph-mon[30184]: pgmap v1684: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-11T21:24:48.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:48 smithi129 ceph-mon[42066]: pgmap v1684: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-11T21:24:48.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:48 smithi157 ceph-mon[33359]: pgmap v1684: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-11T21:24:51.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:50 smithi129 ceph-mon[42066]: pgmap v1685: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-11T21:24:51.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:51.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:51.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:50 smithi129 ceph-mon[30184]: pgmap v1685: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-11T21:24:51.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:51.217 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:51.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:50 smithi157 ceph-mon[33359]: pgmap v1685: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-11T21:24:51.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:24:51.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:24:53.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:52 smithi129 ceph-mon[30184]: pgmap v1686: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-11T21:24:53.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:52 smithi129 ceph-mon[42066]: pgmap v1686: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-11T21:24:53.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:52 smithi157 ceph-mon[33359]: pgmap v1686: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-11T21:24:55.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:54 smithi129 ceph-mon[30184]: pgmap v1687: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:55.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:54 smithi129 ceph-mon[42066]: pgmap v1687: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:55.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:24:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:24:55.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:54 smithi157 ceph-mon[33359]: pgmap v1687: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:24:56.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:56.216 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:24:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:24:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:24:56.217 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:56.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:24:57.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:56 smithi129 ceph-mon[42066]: pgmap v1688: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1008 B/s rd, 0 op/s 2024-03-11T21:24:57.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:56 smithi129 ceph-mon[30184]: pgmap v1688: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1008 B/s rd, 0 op/s 2024-03-11T21:24:57.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:56 smithi157 ceph-mon[33359]: pgmap v1688: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1008 B/s rd, 0 op/s 2024-03-11T21:24:59.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:24:58 smithi129 ceph-mon[30184]: pgmap v1689: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:59.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:24:58 smithi129 ceph-mon[42066]: pgmap v1689: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:24:59.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:24:58 smithi157 ceph-mon[33359]: pgmap v1689: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:01.219 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:00 smithi129 ceph-mon[30184]: pgmap v1690: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:01.220 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:00 smithi129 ceph-mon[42066]: pgmap v1690: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:01.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:00 smithi157 ceph-mon[33359]: pgmap v1690: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:03.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:02 smithi129 ceph-mon[30184]: pgmap v1691: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:03.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:02 smithi129 ceph-mon[42066]: pgmap v1691: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:03.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:02 smithi157 ceph-mon[33359]: pgmap v1691: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:05.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:04 smithi129 ceph-mon[30184]: pgmap v1692: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:05.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:04 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:05.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:04 smithi129 ceph-mon[42066]: pgmap v1692: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:05.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:04 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:05.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:25:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:25:05.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:04 smithi157 ceph-mon[33359]: pgmap v1692: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:05.224 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:04 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:06.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:06.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:06.216 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:25:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:25:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:25:06.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:07.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:06 smithi129 ceph-mon[30184]: pgmap v1693: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:07.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:06 smithi129 ceph-mon[42066]: pgmap v1693: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:07.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:06 smithi157 ceph-mon[33359]: pgmap v1693: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:08.027 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=159224 2024-03-11T21:25:08.027 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph osd pool set-quota ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 max_objects 100 2024-03-11T21:25:08.027 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 put onemore /etc/passwd 2024-03-11T21:25:08.124 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.137+0000 7fe584945640 1 -- 172.21.15.129:0/1168402375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe580105510 msgr2=0x7fe58010d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:08.124 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.137+0000 7fe584945640 1 --2- 172.21.15.129:0/1168402375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe580105510 0x7fe58010d0e0 secure :-1 s=READY pgs=3009 cs=0 l=1 rev1=1 crypto rx=0x7fe578009ca0 tx=0x7fe57802c0f0 comp rx=0 tx=0).stop 2024-03-11T21:25:08.124 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.137+0000 7fe584945640 1 -- 172.21.15.129:0/1168402375 shutdown_connections 2024-03-11T21:25:08.124 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.137+0000 7fe584945640 1 --2- 172.21.15.129:0/1168402375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe580105510 0x7fe58010d0e0 unknown :-1 s=CLOSED pgs=3009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:08.124 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.137+0000 7fe584945640 1 --2- 172.21.15.129:0/1168402375 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe580100df0 0x7fe580104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:08.125 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.137+0000 7fe584945640 1 --2- 172.21.15.129:0/1168402375 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe580100440 0x7fe580100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:08.125 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.137+0000 7fe584945640 1 -- 172.21.15.129:0/1168402375 >> 172.21.15.129:0/1168402375 conn(0x7fe5800773d0 msgr2=0x7fe5800777e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:08.125 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.137+0000 7fe584945640 1 -- 172.21.15.129:0/1168402375 shutdown_connections 2024-03-11T21:25:08.125 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.137+0000 7fe584945640 1 -- 172.21.15.129:0/1168402375 wait complete. 2024-03-11T21:25:08.125 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.138+0000 7fe584945640 1 Processor -- start 2024-03-11T21:25:08.125 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.138+0000 7fe584945640 1 -- start start 2024-03-11T21:25:08.125 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.138+0000 7fe584945640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe580100440 0x7fe5801970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:08.125 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.138+0000 7fe584945640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe580105510 0x7fe580197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:08.125 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.138+0000 7fe584945640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe58019b990 0x7fe58019be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:08.125 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.138+0000 7fe584945640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58010fa30 con 0x7fe580105510 2024-03-11T21:25:08.125 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.138+0000 7fe584945640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58010f8b0 con 0x7fe580100440 2024-03-11T21:25:08.126 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.138+0000 7fe584945640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58010fbb0 con 0x7fe58019b990 2024-03-11T21:25:08.126 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe57ed9d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe58019b990 0x7fe58019be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:08.126 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe57dd9b640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe580105510 0x7fe580197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:08.126 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe57ed9d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe58019b990 0x7fe58019be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:36830/0 (socket says 172.21.15.129:36830) 2024-03-11T21:25:08.126 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe57ed9d640 1 -- 172.21.15.129:0/413278611 learned_addr learned my addr 172.21.15.129:0/413278611 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T21:25:08.126 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe57e59c640 1 --2- 172.21.15.129:0/413278611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe580100440 0x7fe5801970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:08.126 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe57dd9b640 1 -- 172.21.15.129:0/413278611 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe58019b990 msgr2=0x7fe58019be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:08.126 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe57dd9b640 1 --2- 172.21.15.129:0/413278611 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe58019b990 0x7fe58019be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:08.126 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe57dd9b640 1 -- 172.21.15.129:0/413278611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe580100440 msgr2=0x7fe5801970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:08.126 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe57dd9b640 1 --2- 172.21.15.129:0/413278611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe580100440 0x7fe5801970d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:08.126 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe57dd9b640 1 -- 172.21.15.129:0/413278611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58019c480 con 0x7fe580105510 2024-03-11T21:25:08.126 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe57ed9d640 1 --2- 172.21.15.129:0/413278611 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe58019b990 0x7fe58019be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:25:08.126 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe57dd9b640 1 --2- 172.21.15.129:0/413278611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe580105510 0x7fe580197610 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7fe57400e9c0 tx=0x7fe57400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:08.127 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe56f7fe640 1 -- 172.21.15.129:0/413278611 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57400cd80 con 0x7fe580105510 2024-03-11T21:25:08.127 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe56f7fe640 1 -- 172.21.15.129:0/413278611 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe5740035f0 con 0x7fe580105510 2024-03-11T21:25:08.127 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe584945640 1 -- 172.21.15.129:0/413278611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58019c770 con 0x7fe580105510 2024-03-11T21:25:08.127 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe56f7fe640 1 -- 172.21.15.129:0/413278611 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5740044a0 con 0x7fe580105510 2024-03-11T21:25:08.127 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.139+0000 7fe584945640 1 -- 172.21.15.129:0/413278611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5800fe270 con 0x7fe580105510 2024-03-11T21:25:08.128 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.140+0000 7fe584945640 1 -- 172.21.15.129:0/413278611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe580068620 con 0x7fe580105510 2024-03-11T21:25:08.128 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.141+0000 7fe56f7fe640 1 -- 172.21.15.129:0/413278611 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fe574004000 con 0x7fe580105510 2024-03-11T21:25:08.128 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.141+0000 7fe56f7fe640 1 --2- 172.21.15.129:0/413278611 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fe56806ddf0 0x7fe5680702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:08.128 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.141+0000 7fe56f7fe640 1 -- 172.21.15.129:0/413278611 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(770..770 src has 254..770) v4 ==== 9019+0+0 (secure 0 0 0) 0x7fe57408eaf0 con 0x7fe580105510 2024-03-11T21:25:08.128 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.141+0000 7fe57e59c640 1 --2- 172.21.15.129:0/413278611 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fe56806ddf0 0x7fe5680702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:08.128 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.141+0000 7fe56f7fe640 1 -- 172.21.15.129:0/413278611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=771}) v3 -- 0x7fe5680797e0 con 0x7fe580105510 2024-03-11T21:25:08.128 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.142+0000 7fe57e59c640 1 --2- 172.21.15.129:0/413278611 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fe56806ddf0 0x7fe5680702b0 secure :-1 s=READY pgs=4036 cs=0 l=1 rev1=1 crypto rx=0x7fe570005fd0 tx=0x7fe570009470 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:08.130 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.146+0000 7fe56f7fe640 1 -- 172.21.15.129:0/413278611 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fe57405b9a0 con 0x7fe580105510 2024-03-11T21:25:08.305 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.320+0000 7fe584945640 1 -- 172.21.15.129:0/413278611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7fe580104fa0 con 0x7fe580105510 2024-03-11T21:25:08.924 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.940+0000 7fe56f7fe640 1 -- 172.21.15.129:0/413278611 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 v771) v1 ==== 225+0+0 (secure 0 0 0) 0x7fe57405f3f0 con 0x7fe580105510 2024-03-11T21:25:08.925 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.940+0000 7fe56f7fe640 1 -- 172.21.15.129:0/413278611 <== mon.0 v2:172.21.15.129:3300/0 8 ==== osd_map(771..771 src has 254..771) v4 ==== 628+0+0 (secure 0 0 0) 0x7fe574059340 con 0x7fe580105510 2024-03-11T21:25:08.925 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:08.940+0000 7fe56f7fe640 1 -- 172.21.15.129:0/413278611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=772}) v3 -- 0x7fe56807a700 con 0x7fe580105510 2024-03-11T21:25:09.035 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:09.051+0000 7fe584945640 1 -- 172.21.15.129:0/413278611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7fe580100df0 con 0x7fe580105510 2024-03-11T21:25:09.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:08 smithi129 ceph-mon[30184]: pgmap v1694: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:09.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:08 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/413278611' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "100"}]: dispatch 2024-03-11T21:25:09.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:08 smithi129 ceph-mon[42066]: pgmap v1694: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:09.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:08 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/413278611' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "100"}]: dispatch 2024-03-11T21:25:09.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:08 smithi157 ceph-mon[33359]: pgmap v1694: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:09.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:08 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/413278611' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "100"}]: dispatch 2024-03-11T21:25:09.930 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:09.945+0000 7fe56f7fe640 1 -- 172.21.15.129:0/413278611 <== mon.0 v2:172.21.15.129:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 v772) v1 ==== 225+0+0 (secure 0 0 0) 0x7fe5740639b0 con 0x7fe580105510 2024-03-11T21:25:09.930 INFO:tasks.workunit.client.0.smithi129.stderr:set-quota max_objects = 100 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 2024-03-11T21:25:09.932 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:09.948+0000 7fe56f7fe640 1 -- 172.21.15.129:0/413278611 <== mon.0 v2:172.21.15.129:3300/0 10 ==== osd_map(772..772 src has 254..772) v4 ==== 628+0+0 (secure 0 0 0) 0x7fe57408c3c0 con 0x7fe580105510 2024-03-11T21:25:09.932 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:09.948+0000 7fe56f7fe640 1 -- 172.21.15.129:0/413278611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=773}) v3 -- 0x7fe56807a910 con 0x7fe580105510 2024-03-11T21:25:09.933 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:09.949+0000 7fe584945640 1 -- 172.21.15.129:0/413278611 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fe56806ddf0 msgr2=0x7fe5680702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:09.933 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:09.949+0000 7fe584945640 1 --2- 172.21.15.129:0/413278611 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fe56806ddf0 0x7fe5680702b0 secure :-1 s=READY pgs=4036 cs=0 l=1 rev1=1 crypto rx=0x7fe570005fd0 tx=0x7fe570009470 comp rx=0 tx=0).stop 2024-03-11T21:25:09.933 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:09.949+0000 7fe584945640 1 -- 172.21.15.129:0/413278611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe580105510 msgr2=0x7fe580197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:09.934 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:09.949+0000 7fe584945640 1 --2- 172.21.15.129:0/413278611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe580105510 0x7fe580197610 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7fe57400e9c0 tx=0x7fe57400ee80 comp rx=0 tx=0).stop 2024-03-11T21:25:09.934 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:09.949+0000 7fe584945640 1 -- 172.21.15.129:0/413278611 shutdown_connections 2024-03-11T21:25:09.934 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:09.949+0000 7fe584945640 1 --2- 172.21.15.129:0/413278611 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fe56806ddf0 0x7fe5680702b0 unknown :-1 s=CLOSED pgs=4036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:09.934 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:09.949+0000 7fe584945640 1 --2- 172.21.15.129:0/413278611 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe58019b990 0x7fe58019be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:09.934 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:09.949+0000 7fe584945640 1 --2- 172.21.15.129:0/413278611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe580105510 0x7fe580197610 unknown :-1 s=CLOSED pgs=3010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:09.934 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:09.949+0000 7fe584945640 1 --2- 172.21.15.129:0/413278611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe580100440 0x7fe5801970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:09.934 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:09.949+0000 7fe584945640 1 -- 172.21.15.129:0/413278611 >> 172.21.15.129:0/413278611 conn(0x7fe5800773d0 msgr2=0x7fe580102170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:09.934 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:09.949+0000 7fe584945640 1 -- 172.21.15.129:0/413278611 shutdown_connections 2024-03-11T21:25:09.934 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:09.949+0000 7fe584945640 1 -- 172.21.15.129:0/413278611 wait complete. 2024-03-11T21:25:09.949 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 159224 2024-03-11T21:25:10.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/413278611' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "100"}]': finished 2024-03-11T21:25:10.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:09 smithi129 ceph-mon[42066]: osdmap e771: 8 total, 8 up, 8 in 2024-03-11T21:25:10.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:09 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/413278611' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "100"}]: dispatch 2024-03-11T21:25:10.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/413278611' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "100"}]': finished 2024-03-11T21:25:10.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:09 smithi129 ceph-mon[30184]: osdmap e771: 8 total, 8 up, 8 in 2024-03-11T21:25:10.216 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:09 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/413278611' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "100"}]: dispatch 2024-03-11T21:25:10.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/413278611' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "100"}]': finished 2024-03-11T21:25:10.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:09 smithi157 ceph-mon[33359]: osdmap e771: 8 total, 8 up, 8 in 2024-03-11T21:25:10.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:09 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/413278611' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "100"}]: dispatch 2024-03-11T21:25:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 0 -ne 0 ']' 2024-03-11T21:25:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:+ true 2024-03-11T21:25:10.266 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 put twomore /etc/passwd 2024-03-11T21:25:10.305 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph osd pool set-quota ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 max_bytes 100 2024-03-11T21:25:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.407+0000 7f5210832640 1 -- 172.21.15.129:0/1654498224 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c100550 msgr2=0x7f520c10c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.407+0000 7f5210832640 1 --2- 172.21.15.129:0/1654498224 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c100550 0x7f520c10c820 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7f5204009ca0 tx=0x7f520402c0f0 comp rx=0 tx=0).stop 2024-03-11T21:25:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.407+0000 7f5210832640 1 -- 172.21.15.129:0/1654498224 shutdown_connections 2024-03-11T21:25:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.407+0000 7f5210832640 1 --2- 172.21.15.129:0/1654498224 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c100550 0x7f520c10c820 unknown :-1 s=CLOSED pgs=3012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.407+0000 7f5210832640 1 --2- 172.21.15.129:0/1654498224 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f520c0ffbb0 0x7f520c100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:10.394 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.407+0000 7f5210832640 1 --2- 172.21.15.129:0/1654498224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f520c104920 0x7f520c104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.407+0000 7f5210832640 1 -- 172.21.15.129:0/1654498224 >> 172.21.15.129:0/1654498224 conn(0x7f520c075510 msgr2=0x7f520c075920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.407+0000 7f5210832640 1 -- 172.21.15.129:0/1654498224 shutdown_connections 2024-03-11T21:25:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.407+0000 7f5210832640 1 -- 172.21.15.129:0/1654498224 wait complete. 2024-03-11T21:25:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f5210832640 1 Processor -- start 2024-03-11T21:25:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f5210832640 1 -- start start 2024-03-11T21:25:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f5210832640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f520c0ffbb0 0x7f520c197010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f5210832640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c100550 0x7f520c197550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f5210832640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f520c104920 0x7f520c19b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f5210832640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c10e9c0 con 0x7f520c100550 2024-03-11T21:25:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f5210832640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c10e840 con 0x7f520c0ffbb0 2024-03-11T21:25:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f5210832640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520c10eb40 con 0x7f520c104920 2024-03-11T21:25:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f5203fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c100550 0x7f520c197550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:10.395 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f520ad9d640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f520c104920 0x7f520c19b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f5203fff640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c100550 0x7f520c197550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36474/0 (socket says 172.21.15.129:36474) 2024-03-11T21:25:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f5203fff640 1 -- 172.21.15.129:0/4095717130 learned_addr learned my addr 172.21.15.129:0/4095717130 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T21:25:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f520ad9d640 1 -- 172.21.15.129:0/4095717130 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f520c0ffbb0 msgr2=0x7f520c197010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f520a59c640 1 --2- 172.21.15.129:0/4095717130 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f520c0ffbb0 0x7f520c197010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f520ad9d640 1 --2- 172.21.15.129:0/4095717130 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f520c0ffbb0 0x7f520c197010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f520ad9d640 1 -- 172.21.15.129:0/4095717130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c100550 msgr2=0x7f520c197550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f520ad9d640 1 --2- 172.21.15.129:0/4095717130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c100550 0x7f520c197550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f520ad9d640 1 -- 172.21.15.129:0/4095717130 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c19c000 con 0x7f520c104920 2024-03-11T21:25:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.408+0000 7f5203fff640 1 --2- 172.21.15.129:0/4095717130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c100550 0x7f520c197550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:25:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.409+0000 7f520ad9d640 1 --2- 172.21.15.129:0/4095717130 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f520c104920 0x7f520c19b8d0 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f520402c700 tx=0x7f5204032830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.409+0000 7f52037fe640 1 -- 172.21.15.129:0/4095717130 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5204041020 con 0x7f520c104920 2024-03-11T21:25:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.409+0000 7f5210832640 1 -- 172.21.15.129:0/4095717130 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520c19c290 con 0x7f520c104920 2024-03-11T21:25:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.409+0000 7f52037fe640 1 -- 172.21.15.129:0/4095717130 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5204003df0 con 0x7f520c104920 2024-03-11T21:25:10.396 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.409+0000 7f52037fe640 1 -- 172.21.15.129:0/4095717130 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5204039040 con 0x7f520c104920 2024-03-11T21:25:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.409+0000 7f5210832640 1 -- 172.21.15.129:0/4095717130 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f520c1a3b80 con 0x7f520c104920 2024-03-11T21:25:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.410+0000 7f5210832640 1 -- 172.21.15.129:0/4095717130 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f520c068620 con 0x7f520c104920 2024-03-11T21:25:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.410+0000 7f52037fe640 1 -- 172.21.15.129:0/4095717130 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f5204044020 con 0x7f520c104920 2024-03-11T21:25:10.397 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.411+0000 7f52037fe640 1 --2- 172.21.15.129:0/4095717130 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f51f806ddf0 0x7f51f80702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:10.398 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.411+0000 7f52037fe640 1 -- 172.21.15.129:0/4095717130 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(773..773 src has 254..773) v4 ==== 9019+0+0 (secure 0 0 0) 0x7f52040b0940 con 0x7f520c104920 2024-03-11T21:25:10.398 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.411+0000 7f520a59c640 1 --2- 172.21.15.129:0/4095717130 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f51f806ddf0 0x7f51f80702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:10.398 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.411+0000 7f520a59c640 1 --2- 172.21.15.129:0/4095717130 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f51f806ddf0 0x7f51f80702b0 secure :-1 s=READY pgs=4038 cs=0 l=1 rev1=1 crypto rx=0x7f51fc005fd0 tx=0x7f51fc005850 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:10.399 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.415+0000 7f52037fe640 1 -- 172.21.15.129:0/4095717130 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f520404b020 con 0x7f520c104920 2024-03-11T21:25:10.574 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:10.590+0000 7f5210832640 1 -- 172.21.15.129:0/4095717130 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7f520c107e50 con 0x7f520c104920 2024-03-11T21:25:11.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:10 smithi129 ceph-mon[42066]: pgmap v1696: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:11.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/413278611' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "100"}]': finished 2024-03-11T21:25:11.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:10 smithi129 ceph-mon[42066]: osdmap e772: 8 total, 8 up, 8 in 2024-03-11T21:25:11.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:10 smithi129 ceph-mon[42066]: pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' no longer out of quota; removing NO_QUOTA flag 2024-03-11T21:25:11.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:10 smithi129 ceph-mon[42066]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-11T21:25:11.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:10 smithi129 ceph-mon[42066]: osdmap e773: 8 total, 8 up, 8 in 2024-03-11T21:25:11.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:10 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4095717130' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-11T21:25:11.216 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:10 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-11T21:25:11.217 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:10 smithi129 ceph-mon[30184]: pgmap v1696: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:11.217 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/413278611' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "100"}]': finished 2024-03-11T21:25:11.217 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:10 smithi129 ceph-mon[30184]: osdmap e772: 8 total, 8 up, 8 in 2024-03-11T21:25:11.217 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:10 smithi129 ceph-mon[30184]: pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' no longer out of quota; removing NO_QUOTA flag 2024-03-11T21:25:11.217 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:10 smithi129 ceph-mon[30184]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-11T21:25:11.217 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:10 smithi129 ceph-mon[30184]: osdmap e773: 8 total, 8 up, 8 in 2024-03-11T21:25:11.217 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:10 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4095717130' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-11T21:25:11.217 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:10 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-11T21:25:11.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:10 smithi157 ceph-mon[33359]: pgmap v1696: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:11.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/413278611' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "100"}]': finished 2024-03-11T21:25:11.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:10 smithi157 ceph-mon[33359]: osdmap e772: 8 total, 8 up, 8 in 2024-03-11T21:25:11.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:10 smithi157 ceph-mon[33359]: pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' no longer out of quota; removing NO_QUOTA flag 2024-03-11T21:25:11.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:10 smithi157 ceph-mon[33359]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-11T21:25:11.223 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:10 smithi157 ceph-mon[33359]: osdmap e773: 8 total, 8 up, 8 in 2024-03-11T21:25:11.224 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:10 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4095717130' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-11T21:25:11.224 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:10 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-11T21:25:11.257 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:11.273+0000 7f52037fe640 1 -- 172.21.15.129:0/4095717130 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 v774) v1 ==== 221+0+0 (secure 0 0 0) 0x7f520407d7f0 con 0x7f520c104920 2024-03-11T21:25:11.370 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:11.385+0000 7f5210832640 1 -- 172.21.15.129:0/4095717130 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7f520c107a60 con 0x7f520c104920 2024-03-11T21:25:12.265 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:12.280+0000 7f52037fe640 1 -- 172.21.15.129:0/4095717130 <== mon.2 v2:172.21.15.129:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 v775) v1 ==== 221+0+0 (secure 0 0 0) 0x7f5204081240 con 0x7f520c104920 2024-03-11T21:25:12.265 INFO:tasks.workunit.client.0.smithi129.stderr:set-quota max_bytes = 100 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 2024-03-11T21:25:12.267 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:12.283+0000 7f5210832640 1 -- 172.21.15.129:0/4095717130 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f51f806ddf0 msgr2=0x7f51f80702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:12.267 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:12.283+0000 7f5210832640 1 --2- 172.21.15.129:0/4095717130 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f51f806ddf0 0x7f51f80702b0 secure :-1 s=READY pgs=4038 cs=0 l=1 rev1=1 crypto rx=0x7f51fc005fd0 tx=0x7f51fc005850 comp rx=0 tx=0).stop 2024-03-11T21:25:12.267 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:12.283+0000 7f5210832640 1 -- 172.21.15.129:0/4095717130 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f520c104920 msgr2=0x7f520c19b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:12.267 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:12.283+0000 7f5210832640 1 --2- 172.21.15.129:0/4095717130 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f520c104920 0x7f520c19b8d0 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f520402c700 tx=0x7f5204032830 comp rx=0 tx=0).stop 2024-03-11T21:25:12.268 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:12.283+0000 7f5210832640 1 -- 172.21.15.129:0/4095717130 shutdown_connections 2024-03-11T21:25:12.268 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:12.283+0000 7f5210832640 1 --2- 172.21.15.129:0/4095717130 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f51f806ddf0 0x7f51f80702b0 unknown :-1 s=CLOSED pgs=4038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:12.268 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:12.283+0000 7f5210832640 1 --2- 172.21.15.129:0/4095717130 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f520c104920 0x7f520c19b8d0 unknown :-1 s=CLOSED pgs=3086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:12.268 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:12.283+0000 7f5210832640 1 --2- 172.21.15.129:0/4095717130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c100550 0x7f520c197550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:12.268 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:12.283+0000 7f5210832640 1 --2- 172.21.15.129:0/4095717130 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f520c0ffbb0 0x7f520c197010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:12.268 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:12.283+0000 7f5210832640 1 -- 172.21.15.129:0/4095717130 >> 172.21.15.129:0/4095717130 conn(0x7f520c075510 msgr2=0x7f520c0fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:12.268 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:12.283+0000 7f5210832640 1 -- 172.21.15.129:0/4095717130 shutdown_connections 2024-03-11T21:25:12.268 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:12.283+0000 7f5210832640 1 -- 172.21.15.129:0/4095717130 wait complete. 2024-03-11T21:25:12.279 INFO:tasks.workunit.client.0.smithi129.stderr:+ sleep 30 2024-03-11T21:25:12.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]': finished 2024-03-11T21:25:12.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:12 smithi129 ceph-mon[30184]: osdmap e774: 8 total, 8 up, 8 in 2024-03-11T21:25:12.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:12 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4095717130' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-11T21:25:12.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:12 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-11T21:25:12.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:12 smithi129 ceph-mon[30184]: pgmap v1700: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:25:12.716 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]': finished 2024-03-11T21:25:12.716 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:12 smithi129 ceph-mon[42066]: osdmap e774: 8 total, 8 up, 8 in 2024-03-11T21:25:12.716 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:12 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4095717130' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-11T21:25:12.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:12 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-11T21:25:12.717 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:12 smithi129 ceph-mon[42066]: pgmap v1700: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:25:12.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]': finished 2024-03-11T21:25:12.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:12 smithi157 ceph-mon[33359]: osdmap e774: 8 total, 8 up, 8 in 2024-03-11T21:25:12.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:12 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4095717130' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-11T21:25:12.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:12 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-11T21:25:12.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:12 smithi157 ceph-mon[33359]: pgmap v1700: 176 pgs: 176 active+clean; 697 KiB data, 873 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:25:13.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:13 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]': finished 2024-03-11T21:25:13.719 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:13 smithi129 ceph-mon[30184]: osdmap e775: 8 total, 8 up, 8 in 2024-03-11T21:25:13.719 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:13 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]': finished 2024-03-11T21:25:13.720 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:13 smithi129 ceph-mon[42066]: osdmap e775: 8 total, 8 up, 8 in 2024-03-11T21:25:13.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:13 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "100"}]': finished 2024-03-11T21:25:13.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:13 smithi157 ceph-mon[33359]: osdmap e775: 8 total, 8 up, 8 in 2024-03-11T21:25:14.624 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:14 smithi157 ceph-mon[33359]: pgmap v1702: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-11T21:25:14.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:14 smithi129 ceph-mon[30184]: pgmap v1702: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-11T21:25:14.716 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:14 smithi129 ceph-mon[42066]: pgmap v1702: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-11T21:25:15.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:25:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:25:15.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:15.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:15 smithi129 ceph-mon[30184]: pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' is full (reached quota's max_bytes: 100 B) 2024-03-11T21:25:15.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:15 smithi129 ceph-mon[30184]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-11T21:25:15.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:15 smithi129 ceph-mon[30184]: osdmap e776: 8 total, 8 up, 8 in 2024-03-11T21:25:15.716 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:15.716 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:15 smithi129 ceph-mon[42066]: pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' is full (reached quota's max_bytes: 100 B) 2024-03-11T21:25:15.716 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:15 smithi129 ceph-mon[42066]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-11T21:25:15.716 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:15 smithi129 ceph-mon[42066]: osdmap e776: 8 total, 8 up, 8 in 2024-03-11T21:25:15.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:15.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:15 smithi157 ceph-mon[33359]: pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' is full (reached quota's max_bytes: 100 B) 2024-03-11T21:25:15.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:15 smithi157 ceph-mon[33359]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-11T21:25:15.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:15 smithi157 ceph-mon[33359]: osdmap e776: 8 total, 8 up, 8 in 2024-03-11T21:25:16.466 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:16 smithi129 ceph-mon[30184]: pgmap v1704: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 992 B/s rd, 4.7 KiB/s wr, 1 op/s 2024-03-11T21:25:16.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:16 smithi129 ceph-mon[42066]: pgmap v1704: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 992 B/s rd, 4.7 KiB/s wr, 1 op/s 2024-03-11T21:25:16.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:25:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:25:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:25:16.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:16 smithi157 ceph-mon[33359]: pgmap v1704: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 992 B/s rd, 4.7 KiB/s wr, 1 op/s 2024-03-11T21:25:18.715 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:18 smithi129 ceph-mon[30184]: pgmap v1705: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 3.9 KiB/s wr, 1 op/s 2024-03-11T21:25:18.716 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:18 smithi129 ceph-mon[42066]: pgmap v1705: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 3.9 KiB/s wr, 1 op/s 2024-03-11T21:25:18.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:18 smithi157 ceph-mon[33359]: pgmap v1705: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 3.9 KiB/s wr, 1 op/s 2024-03-11T21:25:20.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:20 smithi157 ceph-mon[33359]: pgmap v1706: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-11T21:25:20.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:20.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:20 smithi129 ceph-mon[30184]: pgmap v1706: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-11T21:25:20.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:20.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:20 smithi129 ceph-mon[42066]: pgmap v1706: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-11T21:25:20.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:22.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:22 smithi157 ceph-mon[33359]: pgmap v1707: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 559 B/s rd, 0 op/s 2024-03-11T21:25:22.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:22 smithi129 ceph-mon[30184]: pgmap v1707: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 559 B/s rd, 0 op/s 2024-03-11T21:25:22.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:22 smithi129 ceph-mon[42066]: pgmap v1707: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 559 B/s rd, 0 op/s 2024-03-11T21:25:24.938 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:24 smithi157 ceph-mon[33359]: pgmap v1708: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:24.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:24 smithi129 ceph-mon[30184]: pgmap v1708: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:24.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:24 smithi129 ceph-mon[42066]: pgmap v1708: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:25:25.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:25:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:25:25.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:25.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:25.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:26.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:25:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:25:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:25:26.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:26 smithi129 ceph-mon[30184]: pgmap v1709: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-11T21:25:26.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:26 smithi129 ceph-mon[42066]: pgmap v1709: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-11T21:25:26.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:26 smithi157 ceph-mon[33359]: pgmap v1709: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-11T21:25:28.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:28 smithi129 ceph-mon[30184]: pgmap v1710: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:28.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:28 smithi129 ceph-mon[42066]: pgmap v1710: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:28.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:28 smithi157 ceph-mon[33359]: pgmap v1710: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:30.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:30 smithi129 ceph-mon[30184]: pgmap v1711: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:30.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:30 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:30.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:30 smithi129 ceph-mon[42066]: pgmap v1711: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:30.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:30 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:30.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:30 smithi157 ceph-mon[33359]: pgmap v1711: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:30.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:30 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:31.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:31 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:31.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:31 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:31.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:31 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:25:31.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:31 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:31.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:31 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:31.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:31 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:25:31.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:31 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:31.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:31 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:31.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:31 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:25:32.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:32 smithi129 ceph-mon[30184]: pgmap v1712: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:32.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:32 smithi129 ceph-mon[42066]: pgmap v1712: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:32.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:32 smithi157 ceph-mon[33359]: pgmap v1712: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:34.950 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:34 smithi157 ceph-mon[33359]: pgmap v1713: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:34.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:34 smithi129 ceph-mon[30184]: pgmap v1713: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:34.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:34 smithi129 ceph-mon[42066]: pgmap v1713: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:35.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:25:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:25:35.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:35 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:35.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:35.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:35 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:35.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:35.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:35 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:35.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:36.466 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:25:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:25:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:25:36.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:36 smithi129 ceph-mon[30184]: pgmap v1714: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:36.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:36 smithi129 ceph-mon[42066]: pgmap v1714: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:36.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:36 smithi157 ceph-mon[33359]: pgmap v1714: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:38.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:38 smithi129 ceph-mon[30184]: pgmap v1715: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:38.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:38 smithi129 ceph-mon[42066]: pgmap v1715: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:38.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:38 smithi157 ceph-mon[33359]: pgmap v1715: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:40.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:40 smithi129 ceph-mon[30184]: pgmap v1716: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:40.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:40 smithi129 ceph-mon[42066]: pgmap v1716: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:40.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:40 smithi157 ceph-mon[33359]: pgmap v1716: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:42.281 INFO:tasks.workunit.client.0.smithi129.stderr:+ pid=159440 2024-03-11T21:25:42.281 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph osd pool set-quota ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 max_bytes 0 2024-03-11T21:25:42.281 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 put two /etc/passwd 2024-03-11T21:25:42.378 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.392+0000 7f882b655640 1 -- 172.21.15.129:0/3858863977 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8824100df0 msgr2=0x7f8824104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:42.378 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.392+0000 7f882b655640 1 --2- 172.21.15.129:0/3858863977 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8824100df0 0x7f8824104e30 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7f8814009900 tx=0x7f881402bc30 comp rx=0 tx=0).stop 2024-03-11T21:25:42.378 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.392+0000 7f882b655640 1 -- 172.21.15.129:0/3858863977 shutdown_connections 2024-03-11T21:25:42.378 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.392+0000 7f882b655640 1 --2- 172.21.15.129:0/3858863977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8824105510 0x7f882410d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:42.378 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.392+0000 7f882b655640 1 --2- 172.21.15.129:0/3858863977 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8824100df0 0x7f8824104e30 unknown :-1 s=CLOSED pgs=3014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:42.378 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.392+0000 7f882b655640 1 --2- 172.21.15.129:0/3858863977 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8824100440 0x7f8824100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:42.378 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.392+0000 7f882b655640 1 -- 172.21.15.129:0/3858863977 >> 172.21.15.129:0/3858863977 conn(0x7f88240773d0 msgr2=0x7f88240777e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:42.378 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.392+0000 7f882b655640 1 -- 172.21.15.129:0/3858863977 shutdown_connections 2024-03-11T21:25:42.378 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.392+0000 7f882b655640 1 -- 172.21.15.129:0/3858863977 wait complete. 2024-03-11T21:25:42.378 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.393+0000 7f882b655640 1 Processor -- start 2024-03-11T21:25:42.379 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.393+0000 7f882b655640 1 -- start start 2024-03-11T21:25:42.379 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.393+0000 7f882b655640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8824100440 0x7f8824196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:42.379 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.393+0000 7f882b655640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8824105510 0x7f8824196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:42.379 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.393+0000 7f882b655640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f882419a8e0 0x7f882419ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:42.379 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.393+0000 7f882b655640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882410f9f0 con 0x7f8824105510 2024-03-11T21:25:42.379 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.393+0000 7f882b655640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882410f870 con 0x7f882419a8e0 2024-03-11T21:25:42.379 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.393+0000 7f882b655640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882410fb70 con 0x7f8824100440 2024-03-11T21:25:42.379 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.393+0000 7f8828bf0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8824105510 0x7f8824196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:42.379 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.393+0000 7f88293f1640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8824100440 0x7f8824196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:42.379 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.393+0000 7f8828bf0640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8824105510 0x7f8824196560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45278/0 (socket says 172.21.15.129:45278) 2024-03-11T21:25:42.379 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.393+0000 7f8828bf0640 1 -- 172.21.15.129:0/2869540126 learned_addr learned my addr 172.21.15.129:0/2869540126 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T21:25:42.379 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.394+0000 7f8828bf0640 1 -- 172.21.15.129:0/2869540126 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8824100440 msgr2=0x7f8824196020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:42.379 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.394+0000 7f8828bf0640 1 --2- 172.21.15.129:0/2869540126 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8824100440 0x7f8824196020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:42.380 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.394+0000 7f8828bf0640 1 -- 172.21.15.129:0/2869540126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f882419a8e0 msgr2=0x7f882419ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:42.380 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.394+0000 7f8829bf2640 1 --2- 172.21.15.129:0/2869540126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f882419a8e0 0x7f882419ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:42.380 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.394+0000 7f8828bf0640 1 --2- 172.21.15.129:0/2869540126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f882419a8e0 0x7f882419ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:42.380 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.394+0000 7f8828bf0640 1 -- 172.21.15.129:0/2869540126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882419b3d0 con 0x7f8824105510 2024-03-11T21:25:42.380 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.394+0000 7f8828bf0640 1 --2- 172.21.15.129:0/2869540126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8824105510 0x7f8824196560 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7f8814000c00 tx=0x7f8814004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:42.380 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.394+0000 7f881a7fc640 1 -- 172.21.15.129:0/2869540126 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8814043020 con 0x7f8824105510 2024-03-11T21:25:42.380 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.394+0000 7f881a7fc640 1 -- 172.21.15.129:0/2869540126 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f8814004a30 con 0x7f8824105510 2024-03-11T21:25:42.380 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.394+0000 7f882b655640 1 -- 172.21.15.129:0/2869540126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f882419b660 con 0x7f8824105510 2024-03-11T21:25:42.380 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.394+0000 7f881a7fc640 1 -- 172.21.15.129:0/2869540126 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8814003890 con 0x7f8824105510 2024-03-11T21:25:42.380 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.394+0000 7f882b655640 1 -- 172.21.15.129:0/2869540126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8824069ce0 con 0x7f8824105510 2024-03-11T21:25:42.381 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.395+0000 7f882b655640 1 -- 172.21.15.129:0/2869540126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8824068620 con 0x7f8824105510 2024-03-11T21:25:42.381 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.395+0000 7f881a7fc640 1 -- 172.21.15.129:0/2869540126 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f8814046020 con 0x7f8824105510 2024-03-11T21:25:42.381 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.396+0000 7f881a7fc640 1 --2- 172.21.15.129:0/2869540126 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f881006ddf0 0x7f88100702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:42.381 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.396+0000 7f88293f1640 1 --2- 172.21.15.129:0/2869540126 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f881006ddf0 0x7f88100702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:42.382 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.396+0000 7f881a7fc640 1 -- 172.21.15.129:0/2869540126 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(776..776 src has 254..776) v4 ==== 9019+0+0 (secure 0 0 0) 0x7f88140b0a40 con 0x7f8824105510 2024-03-11T21:25:42.382 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.396+0000 7f881a7fc640 1 -- 172.21.15.129:0/2869540126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=777}) v3 -- 0x7f88100797e0 con 0x7f8824105510 2024-03-11T21:25:42.382 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.396+0000 7f88293f1640 1 --2- 172.21.15.129:0/2869540126 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f881006ddf0 0x7f88100702b0 secure :-1 s=READY pgs=4040 cs=0 l=1 rev1=1 crypto rx=0x7f88200095f0 tx=0x7f8820010040 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:42.384 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.401+0000 7f881a7fc640 1 -- 172.21.15.129:0/2869540126 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f88140b6020 con 0x7f8824105510 2024-03-11T21:25:42.560 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:42.576+0000 7f882b655640 1 -- 172.21.15.129:0/2869540126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f8824100df0 con 0x7f8824105510 2024-03-11T21:25:42.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:42 smithi129 ceph-mon[30184]: pgmap v1717: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:42.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:42 smithi129 ceph-mon[42066]: pgmap v1717: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:42.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:42 smithi157 ceph-mon[33359]: pgmap v1717: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:43.538 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:43.554+0000 7f881a7fc640 1 -- 172.21.15.129:0/2869540126 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 v777) v1 ==== 217+0+0 (secure 0 0 0) 0x7f881404d020 con 0x7f8824105510 2024-03-11T21:25:43.538 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:43.554+0000 7f881a7fc640 1 -- 172.21.15.129:0/2869540126 <== mon.0 v2:172.21.15.129:3300/0 8 ==== osd_map(777..777 src has 254..777) v4 ==== 628+0+0 (secure 0 0 0) 0x7f881409d0b0 con 0x7f8824105510 2024-03-11T21:25:43.538 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:43.554+0000 7f881a7fc640 1 -- 172.21.15.129:0/2869540126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=778}) v3 -- 0x7f881007a700 con 0x7f8824105510 2024-03-11T21:25:43.648 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:43.665+0000 7f882b655640 1 -- 172.21.15.129:0/2869540126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f8824069f70 con 0x7f8824105510 2024-03-11T21:25:43.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:43 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2869540126' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-11T21:25:43.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:43 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2869540126' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-11T21:25:43.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:43 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2869540126' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-11T21:25:44.539 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.556+0000 7f881a7fc640 1 -- 172.21.15.129:0/2869540126 <== mon.0 v2:172.21.15.129:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 v778) v1 ==== 217+0+0 (secure 0 0 0) 0x7f8814005450 con 0x7f8824105510 2024-03-11T21:25:44.539 INFO:tasks.workunit.client.0.smithi129.stderr:set-quota max_bytes = 0 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 2024-03-11T21:25:44.540 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.556+0000 7f881a7fc640 1 -- 172.21.15.129:0/2869540126 <== mon.0 v2:172.21.15.129:3300/0 10 ==== osd_map(778..778 src has 254..778) v4 ==== 628+0+0 (secure 0 0 0) 0x7f881407b290 con 0x7f8824105510 2024-03-11T21:25:44.540 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.556+0000 7f881a7fc640 1 -- 172.21.15.129:0/2869540126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=779}) v3 -- 0x7f881007a7f0 con 0x7f8824105510 2024-03-11T21:25:44.543 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.559+0000 7f882b655640 1 -- 172.21.15.129:0/2869540126 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f881006ddf0 msgr2=0x7f88100702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:44.543 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.559+0000 7f882b655640 1 --2- 172.21.15.129:0/2869540126 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f881006ddf0 0x7f88100702b0 secure :-1 s=READY pgs=4040 cs=0 l=1 rev1=1 crypto rx=0x7f88200095f0 tx=0x7f8820010040 comp rx=0 tx=0).stop 2024-03-11T21:25:44.543 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.559+0000 7f882b655640 1 -- 172.21.15.129:0/2869540126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8824105510 msgr2=0x7f8824196560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:44.543 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.559+0000 7f882b655640 1 --2- 172.21.15.129:0/2869540126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8824105510 0x7f8824196560 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7f8814000c00 tx=0x7f8814004270 comp rx=0 tx=0).stop 2024-03-11T21:25:44.543 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.559+0000 7f882b655640 1 -- 172.21.15.129:0/2869540126 shutdown_connections 2024-03-11T21:25:44.543 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.559+0000 7f882b655640 1 --2- 172.21.15.129:0/2869540126 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f881006ddf0 0x7f88100702b0 unknown :-1 s=CLOSED pgs=4040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.543 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.559+0000 7f882b655640 1 --2- 172.21.15.129:0/2869540126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f882419a8e0 0x7f882419ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.543 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.559+0000 7f882b655640 1 --2- 172.21.15.129:0/2869540126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8824105510 0x7f8824196560 unknown :-1 s=CLOSED pgs=3015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.543 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.560+0000 7f882b655640 1 --2- 172.21.15.129:0/2869540126 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f8824100440 0x7f8824196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.543 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.560+0000 7f882b655640 1 -- 172.21.15.129:0/2869540126 >> 172.21.15.129:0/2869540126 conn(0x7f88240773d0 msgr2=0x7f88240fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:44.544 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.560+0000 7f882b655640 1 -- 172.21.15.129:0/2869540126 shutdown_connections 2024-03-11T21:25:44.544 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.560+0000 7f882b655640 1 -- 172.21.15.129:0/2869540126 wait complete. 2024-03-11T21:25:44.559 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph osd pool set-quota ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 max_objects 0 2024-03-11T21:25:44.649 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.663+0000 7fe790012640 1 -- 172.21.15.129:0/1143648115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe78810b8a0 msgr2=0x7fe78810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:44.649 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.663+0000 7fe790012640 1 --2- 172.21.15.129:0/1143648115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe78810b8a0 0x7fe78810dd40 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7fe780009ca0 tx=0x7fe78002c230 comp rx=0 tx=0).stop 2024-03-11T21:25:44.650 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.663+0000 7fe790012640 1 -- 172.21.15.129:0/1143648115 shutdown_connections 2024-03-11T21:25:44.650 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.663+0000 7fe790012640 1 --2- 172.21.15.129:0/1143648115 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe78810b8a0 0x7fe78810dd40 unknown :-1 s=CLOSED pgs=3016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.650 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.663+0000 7fe790012640 1 --2- 172.21.15.129:0/1143648115 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe788102e80 0x7fe78810b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.650 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.663+0000 7fe790012640 1 --2- 172.21.15.129:0/1143648115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe788102560 0x7fe788102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.650 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.663+0000 7fe790012640 1 -- 172.21.15.129:0/1143648115 >> 172.21.15.129:0/1143648115 conn(0x7fe788077310 msgr2=0x7fe788077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:44.650 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.663+0000 7fe790012640 1 -- 172.21.15.129:0/1143648115 shutdown_connections 2024-03-11T21:25:44.650 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.663+0000 7fe790012640 1 -- 172.21.15.129:0/1143648115 wait complete. 2024-03-11T21:25:44.650 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.663+0000 7fe790012640 1 Processor -- start 2024-03-11T21:25:44.650 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.663+0000 7fe790012640 1 -- start start 2024-03-11T21:25:44.650 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe790012640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe788102560 0x7fe788197020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:44.650 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe790012640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe788102e80 0x7fe788197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:44.650 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe790012640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe78810b8a0 0x7fe78819b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:44.650 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe790012640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe788110af0 con 0x7fe788102e80 2024-03-11T21:25:44.651 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe790012640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe788110970 con 0x7fe78810b8a0 2024-03-11T21:25:44.651 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe790012640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe788110c70 con 0x7fe788102560 2024-03-11T21:25:44.651 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe78ddae640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe788102560 0x7fe788197020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:44.651 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe78d5ad640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe788102e80 0x7fe788197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:44.651 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe78d5ad640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe788102e80 0x7fe788197560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45304/0 (socket says 172.21.15.129:45304) 2024-03-11T21:25:44.651 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe78ddae640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe788102560 0x7fe788197020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:55350/0 (socket says 172.21.15.129:55350) 2024-03-11T21:25:44.651 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe78d5ad640 1 -- 172.21.15.129:0/4082595707 learned_addr learned my addr 172.21.15.129:0/4082595707 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T21:25:44.651 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe78e5af640 1 --2- 172.21.15.129:0/4082595707 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe78810b8a0 0x7fe78819b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:44.651 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe78ddae640 1 -- 172.21.15.129:0/4082595707 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe78810b8a0 msgr2=0x7fe78819b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:44.651 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe78ddae640 1 --2- 172.21.15.129:0/4082595707 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe78810b8a0 0x7fe78819b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.651 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe78ddae640 1 -- 172.21.15.129:0/4082595707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe788102e80 msgr2=0x7fe788197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:44.651 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe78ddae640 1 --2- 172.21.15.129:0/4082595707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe788102e80 0x7fe788197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:44.651 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe78ddae640 1 -- 172.21.15.129:0/4082595707 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78819bfc0 con 0x7fe788102560 2024-03-11T21:25:44.651 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe78d5ad640 1 --2- 172.21.15.129:0/4082595707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe788102e80 0x7fe788197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:25:44.652 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe78ddae640 1 --2- 172.21.15.129:0/4082595707 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe788102560 0x7fe788197020 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7fe78400c970 tx=0x7fe78400ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:44.652 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe77effd640 1 -- 172.21.15.129:0/4082595707 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784007bb0 con 0x7fe788102560 2024-03-11T21:25:44.652 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe790012640 1 -- 172.21.15.129:0/4082595707 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe78819c2b0 con 0x7fe788102560 2024-03-11T21:25:44.652 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe77effd640 1 -- 172.21.15.129:0/4082595707 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fe784007d50 con 0x7fe788102560 2024-03-11T21:25:44.652 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe77effd640 1 -- 172.21.15.129:0/4082595707 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784018660 con 0x7fe788102560 2024-03-11T21:25:44.652 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.664+0000 7fe790012640 1 -- 172.21.15.129:0/4082595707 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7881a3b40 con 0x7fe788102560 2024-03-11T21:25:44.652 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.665+0000 7fe790012640 1 -- 172.21.15.129:0/4082595707 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7880681d0 con 0x7fe788102560 2024-03-11T21:25:44.652 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.666+0000 7fe77effd640 1 -- 172.21.15.129:0/4082595707 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fe78400f450 con 0x7fe788102560 2024-03-11T21:25:44.652 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.666+0000 7fe77effd640 1 --2- 172.21.15.129:0/4082595707 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fe77406ddf0 0x7fe7740702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:44.652 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.666+0000 7fe77effd640 1 -- 172.21.15.129:0/4082595707 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(778..778 src has 254..778) v4 ==== 9019+0+0 (secure 0 0 0) 0x7fe78408e920 con 0x7fe788102560 2024-03-11T21:25:44.652 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.666+0000 7fe78d5ad640 1 --2- 172.21.15.129:0/4082595707 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fe77406ddf0 0x7fe7740702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:44.652 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.666+0000 7fe77effd640 1 -- 172.21.15.129:0/4082595707 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=779}) v3 -- 0x7fe7740797e0 con 0x7fe788102560 2024-03-11T21:25:44.653 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.667+0000 7fe78d5ad640 1 --2- 172.21.15.129:0/4082595707 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fe77406ddf0 0x7fe7740702b0 secure :-1 s=READY pgs=4041 cs=0 l=1 rev1=1 crypto rx=0x7fe7780095f0 tx=0x7fe778005f90 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:44.654 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.671+0000 7fe77effd640 1 -- 172.21.15.129:0/4082595707 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fe784014030 con 0x7fe788102560 2024-03-11T21:25:44.829 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:44.846+0000 7fe790012640 1 -- 172.21.15.129:0/4082595707 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fe788106130 con 0x7fe788102560 2024-03-11T21:25:44.875 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:44 smithi157 ceph-mon[33359]: pgmap v1718: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:44.875 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2869540126' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]': finished 2024-03-11T21:25:44.875 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:44 smithi157 ceph-mon[33359]: osdmap e777: 8 total, 8 up, 8 in 2024-03-11T21:25:44.875 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:44 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2869540126' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-11T21:25:44.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:44 smithi129 ceph-mon[30184]: pgmap v1718: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:44.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2869540126' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]': finished 2024-03-11T21:25:44.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:44 smithi129 ceph-mon[30184]: osdmap e777: 8 total, 8 up, 8 in 2024-03-11T21:25:44.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:44 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2869540126' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-11T21:25:44.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:44 smithi129 ceph-mon[42066]: pgmap v1718: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:25:44.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2869540126' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]': finished 2024-03-11T21:25:44.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:44 smithi129 ceph-mon[42066]: osdmap e777: 8 total, 8 up, 8 in 2024-03-11T21:25:44.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:44 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2869540126' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-11T21:25:45.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:25:44 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:25:45.257 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:45.274+0000 7fe77effd640 1 -- 172.21.15.129:0/4082595707 <== mon.2 v2:172.21.15.129:3301/0 7 ==== osd_map(779..779 src has 254..779) v4 ==== 628+0+0 (secure 0 0 0) 0x7fe784059170 con 0x7fe788102560 2024-03-11T21:25:45.258 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:45.274+0000 7fe77effd640 1 -- 172.21.15.129:0/4082595707 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=780}) v3 -- 0x7fe77407a310 con 0x7fe788102560 2024-03-11T21:25:45.260 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:45.277+0000 7fe77effd640 1 -- 172.21.15.129:0/4082595707 <== mon.2 v2:172.21.15.129:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 v779) v1 ==== 221+0+0 (secure 0 0 0) 0x7fe78405b7d0 con 0x7fe788102560 2024-03-11T21:25:45.368 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:45.384+0000 7fe790012640 1 -- 172.21.15.129:0/4082595707 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fe788105ba0 con 0x7fe788102560 2024-03-11T21:25:45.901 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2869540126' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]': finished 2024-03-11T21:25:45.901 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:45 smithi157 ceph-mon[33359]: osdmap e778: 8 total, 8 up, 8 in 2024-03-11T21:25:45.901 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4082595707' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:25:45.901 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:45 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:25:45.901 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:45 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:45.901 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:45 smithi157 ceph-mon[33359]: pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' no longer out of quota; removing NO_QUOTA flag 2024-03-11T21:25:45.901 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:45 smithi157 ceph-mon[33359]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-11T21:25:45.902 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:45 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]': finished 2024-03-11T21:25:45.902 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:45 smithi157 ceph-mon[33359]: osdmap e779: 8 total, 8 up, 8 in 2024-03-11T21:25:45.902 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:45 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4082595707' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:25:45.902 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:45 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:25:45.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2869540126' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]': finished 2024-03-11T21:25:45.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[30184]: osdmap e778: 8 total, 8 up, 8 in 2024-03-11T21:25:45.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4082595707' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:25:45.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:25:45.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:45.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[30184]: pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' no longer out of quota; removing NO_QUOTA flag 2024-03-11T21:25:45.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[30184]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-11T21:25:45.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]': finished 2024-03-11T21:25:45.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[30184]: osdmap e779: 8 total, 8 up, 8 in 2024-03-11T21:25:45.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4082595707' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:25:45.967 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:25:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2869540126' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]': finished 2024-03-11T21:25:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[42066]: osdmap e778: 8 total, 8 up, 8 in 2024-03-11T21:25:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4082595707' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:25:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:25:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[42066]: pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' no longer out of quota; removing NO_QUOTA flag 2024-03-11T21:25:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[42066]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-11T21:25:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]': finished 2024-03-11T21:25:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[42066]: osdmap e779: 8 total, 8 up, 8 in 2024-03-11T21:25:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4082595707' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:25:45.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:45 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:25:46.269 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.286+0000 7fe77effd640 1 -- 172.21.15.129:0/4082595707 <== mon.2 v2:172.21.15.129:3301/0 9 ==== osd_map(780..780 src has 254..780) v4 ==== 628+0+0 (secure 0 0 0) 0x7fe78408c1f0 con 0x7fe788102560 2024-03-11T21:25:46.274 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.290+0000 7fe77effd640 1 -- 172.21.15.129:0/4082595707 <== mon.2 v2:172.21.15.129:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 v780) v1 ==== 221+0+0 (secure 0 0 0) 0x7fe78405f220 con 0x7fe788102560 2024-03-11T21:25:46.274 INFO:tasks.workunit.client.0.smithi129.stderr:set-quota max_objects = 0 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 2024-03-11T21:25:46.278 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.294+0000 7fe790012640 1 -- 172.21.15.129:0/4082595707 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fe77406ddf0 msgr2=0x7fe7740702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:46.278 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.294+0000 7fe790012640 1 --2- 172.21.15.129:0/4082595707 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fe77406ddf0 0x7fe7740702b0 secure :-1 s=READY pgs=4041 cs=0 l=1 rev1=1 crypto rx=0x7fe7780095f0 tx=0x7fe778005f90 comp rx=0 tx=0).stop 2024-03-11T21:25:46.278 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.294+0000 7fe790012640 1 -- 172.21.15.129:0/4082595707 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe788102560 msgr2=0x7fe788197020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:46.278 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.294+0000 7fe790012640 1 --2- 172.21.15.129:0/4082595707 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe788102560 0x7fe788197020 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7fe78400c970 tx=0x7fe78400ce30 comp rx=0 tx=0).stop 2024-03-11T21:25:46.278 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.294+0000 7fe790012640 1 -- 172.21.15.129:0/4082595707 shutdown_connections 2024-03-11T21:25:46.278 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.294+0000 7fe790012640 1 --2- 172.21.15.129:0/4082595707 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fe77406ddf0 0x7fe7740702b0 unknown :-1 s=CLOSED pgs=4041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.278 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.294+0000 7fe790012640 1 --2- 172.21.15.129:0/4082595707 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe78810b8a0 0x7fe78819b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.278 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.294+0000 7fe790012640 1 --2- 172.21.15.129:0/4082595707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe788102e80 0x7fe788197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.278 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.294+0000 7fe790012640 1 --2- 172.21.15.129:0/4082595707 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fe788102560 0x7fe788197020 unknown :-1 s=CLOSED pgs=3089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.278 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.294+0000 7fe790012640 1 -- 172.21.15.129:0/4082595707 >> 172.21.15.129:0/4082595707 conn(0x7fe788077310 msgr2=0x7fe788103300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:46.278 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.294+0000 7fe790012640 1 -- 172.21.15.129:0/4082595707 shutdown_connections 2024-03-11T21:25:46.278 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.294+0000 7fe790012640 1 -- 172.21.15.129:0/4082595707 wait complete. 2024-03-11T21:25:46.289 INFO:tasks.workunit.client.0.smithi129.stderr:+ wait 159440 2024-03-11T21:25:46.289 INFO:tasks.workunit.client.0.smithi129.stderr:+ '[' 0 -ne 0 ']' 2024-03-11T21:25:46.289 INFO:tasks.workunit.client.0.smithi129.stderr:+ true 2024-03-11T21:25:46.290 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 put three /etc/passwd 2024-03-11T21:25:46.327 INFO:tasks.workunit.client.0.smithi129.stderr:++ uuidgen 2024-03-11T21:25:46.328 INFO:tasks.workunit.client.0.smithi129.stderr:+ pp=4bf093d5-8685-43fb-a522-cbae7e0b1dcf 2024-03-11T21:25:46.328 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph osd pool create 4bf093d5-8685-43fb-a522-cbae7e0b1dcf 12 2024-03-11T21:25:46.414 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.429+0000 7f14c5f37640 1 -- 172.21.15.129:0/755126898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14c0100440 msgr2=0x7f14c0100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:46.414 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.429+0000 7f14c5f37640 1 --2- 172.21.15.129:0/755126898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14c0100440 0x7f14c0100820 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009860 tx=0x7f14b002bb70 comp rx=0 tx=0).stop 2024-03-11T21:25:46.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.429+0000 7f14c5f37640 1 -- 172.21.15.129:0/755126898 shutdown_connections 2024-03-11T21:25:46.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.429+0000 7f14c5f37640 1 --2- 172.21.15.129:0/755126898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14c0105510 0x7f14c010d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.429+0000 7f14c5f37640 1 --2- 172.21.15.129:0/755126898 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14c0100df0 0x7f14c0104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.429+0000 7f14c5f37640 1 --2- 172.21.15.129:0/755126898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14c0100440 0x7f14c0100820 unknown :-1 s=CLOSED pgs=3018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.429+0000 7f14c5f37640 1 -- 172.21.15.129:0/755126898 >> 172.21.15.129:0/755126898 conn(0x7f14c00773d0 msgr2=0x7f14c00777e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:46.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.429+0000 7f14c5f37640 1 -- 172.21.15.129:0/755126898 shutdown_connections 2024-03-11T21:25:46.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.429+0000 7f14c5f37640 1 -- 172.21.15.129:0/755126898 wait complete. 2024-03-11T21:25:46.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.429+0000 7f14c5f37640 1 Processor -- start 2024-03-11T21:25:46.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14c5f37640 1 -- start start 2024-03-11T21:25:46.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14c5f37640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14c0100440 0x7f14c0197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:46.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14c5f37640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14c0105510 0x7f14c01984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:46.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14c5f37640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14c0198a80 0x7f14c01a3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:46.415 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14c5f37640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c010fa20 con 0x7f14c0100440 2024-03-11T21:25:46.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14c5f37640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c010f8a0 con 0x7f14c0198a80 2024-03-11T21:25:46.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14c5f37640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c010fba0 con 0x7f14c0105510 2024-03-11T21:25:46.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14bf7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14c0100440 0x7f14c0197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:46.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14beffd640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14c0105510 0x7f14c01984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:46.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14bf7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14c0100440 0x7f14c0197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53254/0 (socket says 172.21.15.129:53254) 2024-03-11T21:25:46.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14bf7fe640 1 -- 172.21.15.129:0/3709447632 learned_addr learned my addr 172.21.15.129:0/3709447632 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T21:25:46.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14bffff640 1 --2- 172.21.15.129:0/3709447632 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14c0198a80 0x7f14c01a3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:46.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14bf7fe640 1 -- 172.21.15.129:0/3709447632 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14c0105510 msgr2=0x7f14c01984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:46.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14bf7fe640 1 --2- 172.21.15.129:0/3709447632 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14c0105510 0x7f14c01984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14bf7fe640 1 -- 172.21.15.129:0/3709447632 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14c0198a80 msgr2=0x7f14c01a3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:46.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14bf7fe640 1 --2- 172.21.15.129:0/3709447632 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14c0198a80 0x7f14c01a3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:46.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14bf7fe640 1 -- 172.21.15.129:0/3709447632 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c01a3800 con 0x7f14c0100440 2024-03-11T21:25:46.416 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.430+0000 7f14bf7fe640 1 --2- 172.21.15.129:0/3709447632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14c0100440 0x7f14c0197fb0 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f14b0005e00 tx=0x7f14b00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:46.417 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.431+0000 7f14bcff9640 1 -- 172.21.15.129:0/3709447632 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0041020 con 0x7f14c0100440 2024-03-11T21:25:46.417 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.431+0000 7f14c5f37640 1 -- 172.21.15.129:0/3709447632 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c01a3a90 con 0x7f14c0100440 2024-03-11T21:25:46.417 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.431+0000 7f14bcff9640 1 -- 172.21.15.129:0/3709447632 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f14b002c990 con 0x7f14c0100440 2024-03-11T21:25:46.417 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.431+0000 7f14bcff9640 1 -- 172.21.15.129:0/3709447632 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0039040 con 0x7f14c0100440 2024-03-11T21:25:46.417 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.431+0000 7f14c5f37640 1 -- 172.21.15.129:0/3709447632 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c01a3f80 con 0x7f14c0100440 2024-03-11T21:25:46.418 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.432+0000 7f14c5f37640 1 -- 172.21.15.129:0/3709447632 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c0068620 con 0x7f14c0100440 2024-03-11T21:25:46.418 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.432+0000 7f14bcff9640 1 -- 172.21.15.129:0/3709447632 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f14b0044020 con 0x7f14c0100440 2024-03-11T21:25:46.418 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.432+0000 7f14bcff9640 1 --2- 172.21.15.129:0/3709447632 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f14a806ddf0 0x7f14a80702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:46.418 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.433+0000 7f14bcff9640 1 -- 172.21.15.129:0/3709447632 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(780..780 src has 254..780) v4 ==== 9019+0+0 (secure 0 0 0) 0x7f14b00b0130 con 0x7f14c0100440 2024-03-11T21:25:46.418 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.433+0000 7f14beffd640 1 --2- 172.21.15.129:0/3709447632 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f14a806ddf0 0x7f14a80702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:46.418 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.433+0000 7f14beffd640 1 --2- 172.21.15.129:0/3709447632 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f14a806ddf0 0x7f14a80702b0 secure :-1 s=READY pgs=4043 cs=0 l=1 rev1=1 crypto rx=0x7f14c006a540 tx=0x7f14b4009260 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:46.420 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.437+0000 7f14bcff9640 1 -- 172.21.15.129:0/3709447632 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f14b004b020 con 0x7f14c0100440 2024-03-11T21:25:46.465 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:25:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:25:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:25:46.597 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:46.614+0000 7f14c5f37640 1 -- 172.21.15.129:0/3709447632 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pg_num": 12} v 0) v1 -- 0x7f14c0100df0 con 0x7f14c0100440 2024-03-11T21:25:46.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:46 smithi129 ceph-mon[30184]: pgmap v1722: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:46.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:46 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]': finished 2024-03-11T21:25:46.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:46 smithi129 ceph-mon[30184]: osdmap e780: 8 total, 8 up, 8 in 2024-03-11T21:25:46.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:46 smithi129 ceph-mon[42066]: pgmap v1722: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:46.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:46 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]': finished 2024-03-11T21:25:46.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:46 smithi129 ceph-mon[42066]: osdmap e780: 8 total, 8 up, 8 in 2024-03-11T21:25:46.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:46 smithi157 ceph-mon[33359]: pgmap v1722: 176 pgs: 176 active+clean; 720 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:46.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:46 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]': finished 2024-03-11T21:25:46.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:46 smithi157 ceph-mon[33359]: osdmap e780: 8 total, 8 up, 8 in 2024-03-11T21:25:47.561 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.577+0000 7f14bcff9640 1 -- 172.21.15.129:0/3709447632 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pg_num": 12}]=0 pool '4bf093d5-8685-43fb-a522-cbae7e0b1dcf' created v781) v1 ==== 176+0+0 (secure 0 0 0) 0x7f14b007cfe0 con 0x7f14c0100440 2024-03-11T21:25:47.675 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.690+0000 7f14c5f37640 1 -- 172.21.15.129:0/3709447632 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pg_num": 12} v 0) v1 -- 0x7f14c01a41b0 con 0x7f14c0100440 2024-03-11T21:25:47.675 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.691+0000 7f14bcff9640 1 -- 172.21.15.129:0/3709447632 <== mon.0 v2:172.21.15.129:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pg_num": 12}]=0 pool '4bf093d5-8685-43fb-a522-cbae7e0b1dcf' already exists v781) v1 ==== 183+0+0 (secure 0 0 0) 0x7f14b0080a30 con 0x7f14c0100440 2024-03-11T21:25:47.675 INFO:tasks.workunit.client.0.smithi129.stderr:pool '4bf093d5-8685-43fb-a522-cbae7e0b1dcf' already exists 2024-03-11T21:25:47.677 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.693+0000 7f14c5f37640 1 -- 172.21.15.129:0/3709447632 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f14a806ddf0 msgr2=0x7f14a80702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:47.677 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.693+0000 7f14c5f37640 1 --2- 172.21.15.129:0/3709447632 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f14a806ddf0 0x7f14a80702b0 secure :-1 s=READY pgs=4043 cs=0 l=1 rev1=1 crypto rx=0x7f14c006a540 tx=0x7f14b4009260 comp rx=0 tx=0).stop 2024-03-11T21:25:47.677 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.693+0000 7f14c5f37640 1 -- 172.21.15.129:0/3709447632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14c0100440 msgr2=0x7f14c0197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:47.677 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.693+0000 7f14c5f37640 1 --2- 172.21.15.129:0/3709447632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14c0100440 0x7f14c0197fb0 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f14b0005e00 tx=0x7f14b00042f0 comp rx=0 tx=0).stop 2024-03-11T21:25:47.677 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.693+0000 7f14c5f37640 1 -- 172.21.15.129:0/3709447632 shutdown_connections 2024-03-11T21:25:47.677 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.693+0000 7f14c5f37640 1 --2- 172.21.15.129:0/3709447632 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f14a806ddf0 0x7f14a80702b0 unknown :-1 s=CLOSED pgs=4043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:47.678 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.693+0000 7f14c5f37640 1 --2- 172.21.15.129:0/3709447632 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14c0198a80 0x7f14c01a3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:47.678 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.693+0000 7f14c5f37640 1 --2- 172.21.15.129:0/3709447632 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f14c0105510 0x7f14c01984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:47.678 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.693+0000 7f14c5f37640 1 --2- 172.21.15.129:0/3709447632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14c0100440 0x7f14c0197fb0 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:47.678 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.693+0000 7f14c5f37640 1 -- 172.21.15.129:0/3709447632 >> 172.21.15.129:0/3709447632 conn(0x7f14c00773d0 msgr2=0x7f14c00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:47.678 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.693+0000 7f14c5f37640 1 -- 172.21.15.129:0/3709447632 shutdown_connections 2024-03-11T21:25:47.678 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.693+0000 7f14c5f37640 1 -- 172.21.15.129:0/3709447632 wait complete. 2024-03-11T21:25:47.686 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph osd pool application enable 4bf093d5-8685-43fb-a522-cbae7e0b1dcf rados 2024-03-11T21:25:47.772 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.785+0000 7ffa7ca03640 1 -- 172.21.15.129:0/2544056682 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffa78100c00 msgr2=0x7ffa7810ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:47.772 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.785+0000 7ffa7ca03640 1 --2- 172.21.15.129:0/2544056682 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffa78100c00 0x7ffa7810ced0 secure :-1 s=READY pgs=3092 cs=0 l=1 rev1=1 crypto rx=0x7ffa70009cd0 tx=0x7ffa7002c0f0 comp rx=0 tx=0).stop 2024-03-11T21:25:47.772 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.785+0000 7ffa7ca03640 1 -- 172.21.15.129:0/2544056682 shutdown_connections 2024-03-11T21:25:47.772 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.785+0000 7ffa7ca03640 1 --2- 172.21.15.129:0/2544056682 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffa78100c00 0x7ffa7810ced0 unknown :-1 s=CLOSED pgs=3092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:47.772 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.785+0000 7ffa7ca03640 1 --2- 172.21.15.129:0/2544056682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa78100260 0x7ffa781006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:47.772 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.785+0000 7ffa7ca03640 1 --2- 172.21.15.129:0/2544056682 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa78104fd0 0x7ffa781053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:47.772 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.785+0000 7ffa7ca03640 1 -- 172.21.15.129:0/2544056682 >> 172.21.15.129:0/2544056682 conn(0x7ffa78075b50 msgr2=0x7ffa78075f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:47.773 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.785+0000 7ffa7ca03640 1 -- 172.21.15.129:0/2544056682 shutdown_connections 2024-03-11T21:25:47.773 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.785+0000 7ffa7ca03640 1 -- 172.21.15.129:0/2544056682 wait complete. 2024-03-11T21:25:47.773 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.785+0000 7ffa7ca03640 1 Processor -- start 2024-03-11T21:25:47.773 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7ca03640 1 -- start start 2024-03-11T21:25:47.774 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7ca03640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa78100260 0x7ffa78195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:47.774 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7ca03640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa78100c00 0x7ffa78196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:47.774 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7ca03640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffa78104fd0 0x7ffa7819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:47.774 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7ca03640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7810f840 con 0x7ffa78100c00 2024-03-11T21:25:47.774 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7ca03640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7810f6c0 con 0x7ffa78100260 2024-03-11T21:25:47.774 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7ca03640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7810f9c0 con 0x7ffa78104fd0 2024-03-11T21:25:47.774 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa78100c00 0x7ffa78196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:47.774 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7759e640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffa78104fd0 0x7ffa7819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:47.774 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7659c640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa78100c00 0x7ffa78196310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53284/0 (socket says 172.21.15.129:53284) 2024-03-11T21:25:47.774 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7659c640 1 -- 172.21.15.129:0/1279470625 learned_addr learned my addr 172.21.15.129:0/1279470625 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T21:25:47.774 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa76d9d640 1 --2- 172.21.15.129:0/1279470625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa78100260 0x7ffa78195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:47.774 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7759e640 1 -- 172.21.15.129:0/1279470625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa78100260 msgr2=0x7ffa78195dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:47.774 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7759e640 1 --2- 172.21.15.129:0/1279470625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa78100260 0x7ffa78195dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:47.775 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7759e640 1 -- 172.21.15.129:0/1279470625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa78100c00 msgr2=0x7ffa78196310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:47.775 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7759e640 1 --2- 172.21.15.129:0/1279470625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa78100c00 0x7ffa78196310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:47.775 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7759e640 1 -- 172.21.15.129:0/1279470625 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7819adc0 con 0x7ffa78104fd0 2024-03-11T21:25:47.775 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7659c640 1 --2- 172.21.15.129:0/1279470625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa78100c00 0x7ffa78196310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:25:47.775 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7759e640 1 --2- 172.21.15.129:0/1279470625 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffa78104fd0 0x7ffa7819a690 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 crypto rx=0x7ffa7002c630 tx=0x7ffa70032830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:47.775 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa5ffff640 1 -- 172.21.15.129:0/1279470625 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa70041020 con 0x7ffa78104fd0 2024-03-11T21:25:47.775 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.786+0000 7ffa7ca03640 1 -- 172.21.15.129:0/1279470625 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa7819b050 con 0x7ffa78104fd0 2024-03-11T21:25:47.775 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.787+0000 7ffa7ca03640 1 -- 172.21.15.129:0/1279470625 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa781a2940 con 0x7ffa78104fd0 2024-03-11T21:25:47.775 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.787+0000 7ffa5ffff640 1 -- 172.21.15.129:0/1279470625 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ffa70003df0 con 0x7ffa78104fd0 2024-03-11T21:25:47.775 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.787+0000 7ffa5ffff640 1 -- 172.21.15.129:0/1279470625 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa70039040 con 0x7ffa78104fd0 2024-03-11T21:25:47.775 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.788+0000 7ffa7ca03640 1 -- 172.21.15.129:0/1279470625 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa64005080 con 0x7ffa78104fd0 2024-03-11T21:25:47.776 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.788+0000 7ffa5ffff640 1 -- 172.21.15.129:0/1279470625 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7ffa70044020 con 0x7ffa78104fd0 2024-03-11T21:25:47.776 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.788+0000 7ffa5ffff640 1 --2- 172.21.15.129:0/1279470625 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7ffa6006de90 0x7ffa60070350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:47.776 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.788+0000 7ffa5ffff640 1 -- 172.21.15.129:0/1279470625 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(781..781 src has 254..781) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ffa700b03f0 con 0x7ffa78104fd0 2024-03-11T21:25:47.776 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.788+0000 7ffa76d9d640 1 --2- 172.21.15.129:0/1279470625 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7ffa6006de90 0x7ffa60070350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:47.777 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.789+0000 7ffa76d9d640 1 --2- 172.21.15.129:0/1279470625 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7ffa6006de90 0x7ffa60070350 secure :-1 s=READY pgs=4044 cs=0 l=1 rev1=1 crypto rx=0x7ffa68005fd0 tx=0x7ffa6800b510 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:47.777 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.793+0000 7ffa5ffff640 1 -- 172.21.15.129:0/1279470625 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7ffa7004b030 con 0x7ffa78104fd0 2024-03-11T21:25:47.955 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:47.971+0000 7ffa7ca03640 1 -- 172.21.15.129:0/1279470625 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"} v 0) v1 -- 0x7ffa64005370 con 0x7ffa78104fd0 2024-03-11T21:25:47.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:47 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3709447632' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pg_num": 12}]: dispatch 2024-03-11T21:25:47.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:47 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3709447632' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pg_num": 12}]: dispatch 2024-03-11T21:25:47.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:47 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3709447632' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pg_num": 12}]: dispatch 2024-03-11T21:25:48.578 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:48.594+0000 7ffa5ffff640 1 -- 172.21.15.129:0/1279470625 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]=0 enabled application 'rados' on pool '4bf093d5-8685-43fb-a522-cbae7e0b1dcf' v782) v1 ==== 213+0+0 (secure 0 0 0) 0x7ffa7007d120 con 0x7ffa78104fd0 2024-03-11T21:25:48.692 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:48.708+0000 7ffa7ca03640 1 -- 172.21.15.129:0/1279470625 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"} v 0) v1 -- 0x7ffa64002400 con 0x7ffa78104fd0 2024-03-11T21:25:48.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:48 smithi129 ceph-mon[30184]: pgmap v1724: 176 pgs: 176 active+clean; 743 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-11T21:25:48.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3709447632' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pg_num": 12}]': finished 2024-03-11T21:25:48.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:48 smithi129 ceph-mon[30184]: osdmap e781: 8 total, 8 up, 8 in 2024-03-11T21:25:48.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3709447632' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pg_num": 12}]: dispatch 2024-03-11T21:25:48.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:48 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1279470625' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]: dispatch 2024-03-11T21:25:48.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:48 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]: dispatch 2024-03-11T21:25:48.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:48 smithi129 ceph-mon[42066]: pgmap v1724: 176 pgs: 176 active+clean; 743 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-11T21:25:48.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3709447632' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pg_num": 12}]': finished 2024-03-11T21:25:48.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:48 smithi129 ceph-mon[42066]: osdmap e781: 8 total, 8 up, 8 in 2024-03-11T21:25:48.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3709447632' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pg_num": 12}]: dispatch 2024-03-11T21:25:48.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:48 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1279470625' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]: dispatch 2024-03-11T21:25:48.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:48 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]: dispatch 2024-03-11T21:25:48.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:48 smithi157 ceph-mon[33359]: pgmap v1724: 176 pgs: 176 active+clean; 743 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-11T21:25:48.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3709447632' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pg_num": 12}]': finished 2024-03-11T21:25:48.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:48 smithi157 ceph-mon[33359]: osdmap e781: 8 total, 8 up, 8 in 2024-03-11T21:25:48.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3709447632' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pg_num": 12}]: dispatch 2024-03-11T21:25:48.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:48 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1279470625' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]: dispatch 2024-03-11T21:25:48.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:48 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]: dispatch 2024-03-11T21:25:49.582 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.598+0000 7ffa5ffff640 1 -- 172.21.15.129:0/1279470625 <== mon.2 v2:172.21.15.129:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]=0 enabled application 'rados' on pool '4bf093d5-8685-43fb-a522-cbae7e0b1dcf' v783) v1 ==== 213+0+0 (secure 0 0 0) 0x7ffa64002400 con 0x7ffa78104fd0 2024-03-11T21:25:49.582 INFO:tasks.workunit.client.0.smithi129.stderr:enabled application 'rados' on pool '4bf093d5-8685-43fb-a522-cbae7e0b1dcf' 2024-03-11T21:25:49.585 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.601+0000 7ffa7ca03640 1 -- 172.21.15.129:0/1279470625 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7ffa6006de90 msgr2=0x7ffa60070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:49.585 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.601+0000 7ffa7ca03640 1 --2- 172.21.15.129:0/1279470625 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7ffa6006de90 0x7ffa60070350 secure :-1 s=READY pgs=4044 cs=0 l=1 rev1=1 crypto rx=0x7ffa68005fd0 tx=0x7ffa6800b510 comp rx=0 tx=0).stop 2024-03-11T21:25:49.585 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.601+0000 7ffa7ca03640 1 -- 172.21.15.129:0/1279470625 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffa78104fd0 msgr2=0x7ffa7819a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:49.585 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.601+0000 7ffa7ca03640 1 --2- 172.21.15.129:0/1279470625 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffa78104fd0 0x7ffa7819a690 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 crypto rx=0x7ffa7002c630 tx=0x7ffa70032830 comp rx=0 tx=0).stop 2024-03-11T21:25:49.585 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.601+0000 7ffa7ca03640 1 -- 172.21.15.129:0/1279470625 shutdown_connections 2024-03-11T21:25:49.586 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.601+0000 7ffa7ca03640 1 --2- 172.21.15.129:0/1279470625 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7ffa6006de90 0x7ffa60070350 unknown :-1 s=CLOSED pgs=4044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:49.586 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.601+0000 7ffa7ca03640 1 --2- 172.21.15.129:0/1279470625 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7ffa78104fd0 0x7ffa7819a690 unknown :-1 s=CLOSED pgs=3093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:49.586 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.601+0000 7ffa7ca03640 1 --2- 172.21.15.129:0/1279470625 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa78100c00 0x7ffa78196310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:49.586 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.601+0000 7ffa7ca03640 1 --2- 172.21.15.129:0/1279470625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa78100260 0x7ffa78195dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:49.586 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.601+0000 7ffa7ca03640 1 -- 172.21.15.129:0/1279470625 >> 172.21.15.129:0/1279470625 conn(0x7ffa78075b50 msgr2=0x7ffa78103c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:49.586 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.601+0000 7ffa7ca03640 1 -- 172.21.15.129:0/1279470625 shutdown_connections 2024-03-11T21:25:49.586 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.601+0000 7ffa7ca03640 1 -- 172.21.15.129:0/1279470625 wait complete. 2024-03-11T21:25:49.596 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph osd pool set-quota 4bf093d5-8685-43fb-a522-cbae7e0b1dcf max_objects 10 2024-03-11T21:25:49.687 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.700+0000 7f911ee9a640 1 -- 172.21.15.129:0/876279400 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9118100440 msgr2=0x7f9118100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:49.687 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.700+0000 7f911ee9a640 1 --2- 172.21.15.129:0/876279400 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9118100440 0x7f9118100820 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7f9114009860 tx=0x7f911402bb70 comp rx=0 tx=0).stop 2024-03-11T21:25:49.687 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.700+0000 7f911ee9a640 1 -- 172.21.15.129:0/876279400 shutdown_connections 2024-03-11T21:25:49.687 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.700+0000 7f911ee9a640 1 --2- 172.21.15.129:0/876279400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9118105510 0x7f911810d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:49.687 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.700+0000 7f911ee9a640 1 --2- 172.21.15.129:0/876279400 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9118100df0 0x7f9118104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:49.687 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.700+0000 7f911ee9a640 1 --2- 172.21.15.129:0/876279400 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9118100440 0x7f9118100820 unknown :-1 s=CLOSED pgs=3094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:49.687 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.700+0000 7f911ee9a640 1 -- 172.21.15.129:0/876279400 >> 172.21.15.129:0/876279400 conn(0x7f91180773d0 msgr2=0x7f91180777e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:49.687 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.700+0000 7f911ee9a640 1 -- 172.21.15.129:0/876279400 shutdown_connections 2024-03-11T21:25:49.687 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.700+0000 7f911ee9a640 1 -- 172.21.15.129:0/876279400 wait complete. 2024-03-11T21:25:49.687 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f911ee9a640 1 Processor -- start 2024-03-11T21:25:49.687 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f911ee9a640 1 -- start start 2024-03-11T21:25:49.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f911ee9a640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9118100440 0x7f91181971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:49.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f911ee9a640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9118105510 0x7f9118197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:49.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f911ee9a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f911819ba60 0x7f911819bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:49.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f911ee9a640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911810fa30 con 0x7f9118100440 2024-03-11T21:25:49.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f911ee9a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911810f8b0 con 0x7f911819ba60 2024-03-11T21:25:49.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f911ee9a640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911810fbb0 con 0x7f9118105510 2024-03-11T21:25:49.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f911cc36640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9118100440 0x7f91181971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:49.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f910ffff640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9118105510 0x7f9118197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:49.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f911cc36640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9118100440 0x7f91181971f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53292/0 (socket says 172.21.15.129:53292) 2024-03-11T21:25:49.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f911cc36640 1 -- 172.21.15.129:0/2224343314 learned_addr learned my addr 172.21.15.129:0/2224343314 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T21:25:49.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f911d437640 1 --2- 172.21.15.129:0/2224343314 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f911819ba60 0x7f911819bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:49.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f910ffff640 1 -- 172.21.15.129:0/2224343314 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f911819ba60 msgr2=0x7f911819bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:49.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f910ffff640 1 --2- 172.21.15.129:0/2224343314 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f911819ba60 0x7f911819bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:49.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f910ffff640 1 -- 172.21.15.129:0/2224343314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9118100440 msgr2=0x7f91181971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:49.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f910ffff640 1 --2- 172.21.15.129:0/2224343314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9118100440 0x7f91181971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:49.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f910ffff640 1 -- 172.21.15.129:0/2224343314 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911819c550 con 0x7f9118105510 2024-03-11T21:25:49.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.701+0000 7f911cc36640 1 --2- 172.21.15.129:0/2224343314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9118100440 0x7f91181971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:25:49.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.702+0000 7f910ffff640 1 --2- 172.21.15.129:0/2224343314 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9118105510 0x7f9118197730 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 crypto rx=0x7f910800c940 tx=0x7f910800ce00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:49.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.702+0000 7f910dffb640 1 -- 172.21.15.129:0/2224343314 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9108007bb0 con 0x7f9118105510 2024-03-11T21:25:49.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.702+0000 7f910dffb640 1 -- 172.21.15.129:0/2224343314 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f9108007d50 con 0x7f9118105510 2024-03-11T21:25:49.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.702+0000 7f911ee9a640 1 -- 172.21.15.129:0/2224343314 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911819c840 con 0x7f9118105510 2024-03-11T21:25:49.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.702+0000 7f910dffb640 1 -- 172.21.15.129:0/2224343314 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9108004cc0 con 0x7f9118105510 2024-03-11T21:25:49.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.702+0000 7f911ee9a640 1 -- 172.21.15.129:0/2224343314 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91181a3fe0 con 0x7f9118105510 2024-03-11T21:25:49.690 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.703+0000 7f911ee9a640 1 -- 172.21.15.129:0/2224343314 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9118068620 con 0x7f9118105510 2024-03-11T21:25:49.690 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.703+0000 7f910dffb640 1 -- 172.21.15.129:0/2224343314 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f9108004ea0 con 0x7f9118105510 2024-03-11T21:25:49.690 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.704+0000 7f910dffb640 1 --2- 172.21.15.129:0/2224343314 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f910006ddf0 0x7f91000702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:25:49.691 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.704+0000 7f910dffb640 1 -- 172.21.15.129:0/2224343314 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(783..783 src has 254..783) v4 ==== 9407+0+0 (secure 0 0 0) 0x7f910808e6d0 con 0x7f9118105510 2024-03-11T21:25:49.691 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.704+0000 7f911cc36640 1 --2- 172.21.15.129:0/2224343314 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f910006ddf0 0x7f91000702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:25:49.691 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.704+0000 7f911cc36640 1 --2- 172.21.15.129:0/2224343314 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f910006ddf0 0x7f91000702b0 secure :-1 s=READY pgs=4045 cs=0 l=1 rev1=1 crypto rx=0x7f911402c180 tx=0x7f911402e040 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:25:49.693 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.708+0000 7f910dffb640 1 -- 172.21.15.129:0/2224343314 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f9108014030 con 0x7f9118105510 2024-03-11T21:25:49.869 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:49.885+0000 7f911ee9a640 1 -- 172.21.15.129:0/2224343314 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f9118100df0 con 0x7f9118105510 2024-03-11T21:25:49.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]': finished 2024-03-11T21:25:49.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:49 smithi129 ceph-mon[30184]: osdmap e782: 8 total, 8 up, 8 in 2024-03-11T21:25:49.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:49 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1279470625' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]: dispatch 2024-03-11T21:25:49.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:49 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]: dispatch 2024-03-11T21:25:49.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]': finished 2024-03-11T21:25:49.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:49 smithi129 ceph-mon[42066]: osdmap e782: 8 total, 8 up, 8 in 2024-03-11T21:25:49.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:49 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1279470625' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]: dispatch 2024-03-11T21:25:49.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:49 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]: dispatch 2024-03-11T21:25:49.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]': finished 2024-03-11T21:25:49.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:49 smithi157 ceph-mon[33359]: osdmap e782: 8 total, 8 up, 8 in 2024-03-11T21:25:49.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:49 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1279470625' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]: dispatch 2024-03-11T21:25:49.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:49 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]: dispatch 2024-03-11T21:25:50.582 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:50.598+0000 7f910dffb640 1 -- 172.21.15.129:0/2224343314 <== mon.2 v2:172.21.15.129:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 4bf093d5-8685-43fb-a522-cbae7e0b1dcf v784) v1 ==== 223+0+0 (secure 0 0 0) 0x7f910805ee50 con 0x7f9118105510 2024-03-11T21:25:50.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:50.705+0000 7f911ee9a640 1 -- 172.21.15.129:0/2224343314 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f91181a4270 con 0x7f9118105510 2024-03-11T21:25:50.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:50 smithi129 ceph-mon[30184]: pgmap v1727: 188 pgs: 12 unknown, 176 active+clean; 743 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.8 KiB/s wr, 1 op/s 2024-03-11T21:25:50.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]': finished 2024-03-11T21:25:50.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:50 smithi129 ceph-mon[30184]: osdmap e783: 8 total, 8 up, 8 in 2024-03-11T21:25:50.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:50 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2224343314' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:25:50.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:50 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:25:50.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:25:50.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:50.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:50 smithi129 ceph-mon[42066]: pgmap v1727: 188 pgs: 12 unknown, 176 active+clean; 743 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.8 KiB/s wr, 1 op/s 2024-03-11T21:25:50.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]': finished 2024-03-11T21:25:50.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:50 smithi129 ceph-mon[42066]: osdmap e783: 8 total, 8 up, 8 in 2024-03-11T21:25:50.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:50 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2224343314' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:25:50.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:50 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:25:50.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:25:50.967 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:50.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:50 smithi157 ceph-mon[33359]: pgmap v1727: 188 pgs: 12 unknown, 176 active+clean; 743 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.8 KiB/s wr, 1 op/s 2024-03-11T21:25:50.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "app": "rados"}]': finished 2024-03-11T21:25:50.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:50 smithi157 ceph-mon[33359]: osdmap e783: 8 total, 8 up, 8 in 2024-03-11T21:25:50.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:50 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2224343314' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:25:50.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:50 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:25:50.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:25:50.974 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:25:51.595 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:51.611+0000 7f910dffb640 1 -- 172.21.15.129:0/2224343314 <== mon.2 v2:172.21.15.129:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 4bf093d5-8685-43fb-a522-cbae7e0b1dcf v785) v1 ==== 223+0+0 (secure 0 0 0) 0x7f9108063410 con 0x7f9118105510 2024-03-11T21:25:51.595 INFO:tasks.workunit.client.0.smithi129.stderr:set-quota max_objects = 10 for pool 4bf093d5-8685-43fb-a522-cbae7e0b1dcf 2024-03-11T21:25:51.598 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:51.613+0000 7f911ee9a640 1 -- 172.21.15.129:0/2224343314 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f910006ddf0 msgr2=0x7f91000702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:51.598 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:51.613+0000 7f911ee9a640 1 --2- 172.21.15.129:0/2224343314 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f910006ddf0 0x7f91000702b0 secure :-1 s=READY pgs=4045 cs=0 l=1 rev1=1 crypto rx=0x7f911402c180 tx=0x7f911402e040 comp rx=0 tx=0).stop 2024-03-11T21:25:51.598 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:51.613+0000 7f911ee9a640 1 -- 172.21.15.129:0/2224343314 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9118105510 msgr2=0x7f9118197730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:25:51.598 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:51.613+0000 7f911ee9a640 1 --2- 172.21.15.129:0/2224343314 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9118105510 0x7f9118197730 secure :-1 s=READY pgs=3095 cs=0 l=1 rev1=1 crypto rx=0x7f910800c940 tx=0x7f910800ce00 comp rx=0 tx=0).stop 2024-03-11T21:25:51.598 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:51.613+0000 7f911ee9a640 1 -- 172.21.15.129:0/2224343314 shutdown_connections 2024-03-11T21:25:51.598 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:51.613+0000 7f911ee9a640 1 --2- 172.21.15.129:0/2224343314 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f910006ddf0 0x7f91000702b0 unknown :-1 s=CLOSED pgs=4045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.598 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:51.613+0000 7f911ee9a640 1 --2- 172.21.15.129:0/2224343314 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f911819ba60 0x7f911819bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.598 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:51.613+0000 7f911ee9a640 1 --2- 172.21.15.129:0/2224343314 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f9118105510 0x7f9118197730 unknown :-1 s=CLOSED pgs=3095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.599 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:51.613+0000 7f911ee9a640 1 --2- 172.21.15.129:0/2224343314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9118100440 0x7f91181971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:25:51.599 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:51.614+0000 7f911ee9a640 1 -- 172.21.15.129:0/2224343314 >> 172.21.15.129:0/2224343314 conn(0x7f91180773d0 msgr2=0x7f9118104010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:25:51.599 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:51.614+0000 7f911ee9a640 1 -- 172.21.15.129:0/2224343314 shutdown_connections 2024-03-11T21:25:51.599 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:25:51.614+0000 7f911ee9a640 1 -- 172.21.15.129:0/2224343314 wait complete. 2024-03-11T21:25:51.609 INFO:tasks.workunit.client.0.smithi129.stderr:+ sleep 30 2024-03-11T21:25:51.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:51 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]': finished 2024-03-11T21:25:51.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:51 smithi129 ceph-mon[30184]: osdmap e784: 8 total, 8 up, 8 in 2024-03-11T21:25:51.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:51 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/2224343314' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:25:51.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:51 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:25:51.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:51 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]': finished 2024-03-11T21:25:51.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:51 smithi129 ceph-mon[42066]: osdmap e784: 8 total, 8 up, 8 in 2024-03-11T21:25:51.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:51 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/2224343314' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:25:51.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:51 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:25:51.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:51 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]': finished 2024-03-11T21:25:51.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:51 smithi157 ceph-mon[33359]: osdmap e784: 8 total, 8 up, 8 in 2024-03-11T21:25:51.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:51 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/2224343314' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:25:51.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:51 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-11T21:25:52.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:52 smithi129 ceph-mon[30184]: pgmap v1730: 188 pgs: 12 unknown, 176 active+clean; 743 KiB data, 870 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:25:52.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:52 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]': finished 2024-03-11T21:25:52.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:52 smithi129 ceph-mon[30184]: osdmap e785: 8 total, 8 up, 8 in 2024-03-11T21:25:52.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:52 smithi129 ceph-mon[42066]: pgmap v1730: 188 pgs: 12 unknown, 176 active+clean; 743 KiB data, 870 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:25:52.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:52 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]': finished 2024-03-11T21:25:52.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:52 smithi129 ceph-mon[42066]: osdmap e785: 8 total, 8 up, 8 in 2024-03-11T21:25:52.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:52 smithi157 ceph-mon[33359]: pgmap v1730: 188 pgs: 12 unknown, 176 active+clean; 743 KiB data, 870 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:25:52.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:52 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "field": "max_objects", "val": "10"}]': finished 2024-03-11T21:25:52.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:52 smithi157 ceph-mon[33359]: osdmap e785: 8 total, 8 up, 8 in 2024-03-11T21:25:54.963 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:54 smithi157 ceph-mon[33359]: pgmap v1732: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:25:54.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:54 smithi129 ceph-mon[30184]: pgmap v1732: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:25:54.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:54 smithi129 ceph-mon[42066]: pgmap v1732: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:25:55.223 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:25:54 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:25:55.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:55 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:55.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:55 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:55.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:55 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:25:56.465 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:25:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:25:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:25:56.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:56 smithi129 ceph-mon[30184]: pgmap v1733: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:56.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:56 smithi129 ceph-mon[42066]: pgmap v1733: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:56.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:56 smithi157 ceph-mon[33359]: pgmap v1733: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:25:58.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:25:58 smithi129 ceph-mon[30184]: pgmap v1734: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:25:58.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:25:58 smithi129 ceph-mon[42066]: pgmap v1734: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:25:58.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:25:58 smithi157 ceph-mon[33359]: pgmap v1734: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:26:00.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:00 smithi129 ceph-mon[30184]: pgmap v1735: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:26:00.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:00 smithi129 ceph-mon[42066]: pgmap v1735: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:26:00.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:00 smithi157 ceph-mon[33359]: pgmap v1735: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:26:02.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:02 smithi129 ceph-mon[30184]: pgmap v1736: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:02.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:02 smithi129 ceph-mon[42066]: pgmap v1736: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:02.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:02 smithi157 ceph-mon[33359]: pgmap v1736: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:04.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:04 smithi129 ceph-mon[30184]: pgmap v1737: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:26:04.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:04 smithi129 ceph-mon[42066]: pgmap v1737: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:26:04.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:04 smithi157 ceph-mon[33359]: pgmap v1737: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:26:05.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:26:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:26:05.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:05.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:05 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:05.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:05.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:05 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:05.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:05.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:05 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:06.465 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:26:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:26:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:26:06.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:06 smithi129 ceph-mon[30184]: pgmap v1738: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:06.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:06 smithi129 ceph-mon[42066]: pgmap v1738: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:06.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:06 smithi157 ceph-mon[33359]: pgmap v1738: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:08.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:08 smithi129 ceph-mon[30184]: pgmap v1739: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:08.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:08 smithi129 ceph-mon[42066]: pgmap v1739: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:08.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:08 smithi157 ceph-mon[33359]: pgmap v1739: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:10.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:10 smithi129 ceph-mon[30184]: pgmap v1740: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:10.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:10 smithi129 ceph-mon[42066]: pgmap v1740: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:10.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:10 smithi157 ceph-mon[33359]: pgmap v1740: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:12.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:12 smithi129 ceph-mon[30184]: pgmap v1741: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:12.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:12 smithi129 ceph-mon[42066]: pgmap v1741: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:12.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:12 smithi157 ceph-mon[33359]: pgmap v1741: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:14.965 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:14 smithi129 ceph-mon[42066]: pgmap v1742: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:14.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:14 smithi129 ceph-mon[30184]: pgmap v1742: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:14.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:14 smithi157 ceph-mon[33359]: pgmap v1742: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:15.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:26:15 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:26:15.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:15 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:15.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:15 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:15.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:15 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:16.465 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:26:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:26:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:26:16.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:16 smithi129 ceph-mon[30184]: pgmap v1743: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:26:16.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:16 smithi129 ceph-mon[42066]: pgmap v1743: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:26:16.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:16 smithi157 ceph-mon[33359]: pgmap v1743: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:26:18.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:18 smithi129 ceph-mon[30184]: pgmap v1744: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:18.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:18 smithi129 ceph-mon[42066]: pgmap v1744: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:18.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:18 smithi157 ceph-mon[33359]: pgmap v1744: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:20.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:20 smithi129 ceph-mon[30184]: pgmap v1745: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:26:20.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:20.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:20 smithi129 ceph-mon[42066]: pgmap v1745: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:26:20.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:20.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:20 smithi157 ceph-mon[33359]: pgmap v1745: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:26:20.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:21.611 INFO:tasks.workunit.client.0.smithi129.stderr:++ seq 1 10 2024-03-11T21:26:21.612 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:26:21.612 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p 4bf093d5-8685-43fb-a522-cbae7e0b1dcf put obj1 /etc/passwd 2024-03-11T21:26:21.655 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:26:21.655 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p 4bf093d5-8685-43fb-a522-cbae7e0b1dcf put obj2 /etc/passwd 2024-03-11T21:26:21.692 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:26:21.692 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p 4bf093d5-8685-43fb-a522-cbae7e0b1dcf put obj3 /etc/passwd 2024-03-11T21:26:21.727 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:26:21.728 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p 4bf093d5-8685-43fb-a522-cbae7e0b1dcf put obj4 /etc/passwd 2024-03-11T21:26:21.763 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:26:21.763 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p 4bf093d5-8685-43fb-a522-cbae7e0b1dcf put obj5 /etc/passwd 2024-03-11T21:26:21.797 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:26:21.797 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p 4bf093d5-8685-43fb-a522-cbae7e0b1dcf put obj6 /etc/passwd 2024-03-11T21:26:21.833 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:26:21.833 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p 4bf093d5-8685-43fb-a522-cbae7e0b1dcf put obj7 /etc/passwd 2024-03-11T21:26:21.867 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:26:21.868 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p 4bf093d5-8685-43fb-a522-cbae7e0b1dcf put obj8 /etc/passwd 2024-03-11T21:26:21.902 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:26:21.902 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p 4bf093d5-8685-43fb-a522-cbae7e0b1dcf put obj9 /etc/passwd 2024-03-11T21:26:21.937 INFO:tasks.workunit.client.0.smithi129.stderr:+ for f in `seq 1 10` 2024-03-11T21:26:21.937 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p 4bf093d5-8685-43fb-a522-cbae7e0b1dcf put obj10 /etc/passwd 2024-03-11T21:26:21.971 INFO:tasks.workunit.client.0.smithi129.stderr:+ sleep 30 2024-03-11T21:26:22.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:22 smithi129 ceph-mon[30184]: pgmap v1746: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:22.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:22 smithi129 ceph-mon[42066]: pgmap v1746: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:22.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:22 smithi157 ceph-mon[33359]: pgmap v1746: 188 pgs: 188 active+clean; 743 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:24.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:24 smithi129 ceph-mon[30184]: pgmap v1747: 188 pgs: 188 active+clean; 778 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-11T21:26:24.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:24 smithi129 ceph-mon[42066]: pgmap v1747: 188 pgs: 188 active+clean; 778 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-11T21:26:24.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:24 smithi157 ceph-mon[33359]: pgmap v1747: 188 pgs: 188 active+clean; 778 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-11T21:26:25.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:26:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:26:25.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:25 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:25.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:25 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:25.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:25 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:26.465 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:26:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:26:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:26:26.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:26 smithi129 ceph-mon[30184]: pgmap v1748: 188 pgs: 188 active+clean; 778 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-11T21:26:26.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:26 smithi129 ceph-mon[42066]: pgmap v1748: 188 pgs: 188 active+clean; 778 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-11T21:26:26.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:26 smithi157 ceph-mon[33359]: pgmap v1748: 188 pgs: 188 active+clean; 778 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-11T21:26:28.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:28 smithi129 ceph-mon[30184]: pgmap v1749: 188 pgs: 188 active+clean; 858 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-11T21:26:28.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:28 smithi129 ceph-mon[42066]: pgmap v1749: 188 pgs: 188 active+clean; 858 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-11T21:26:28.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:28 smithi157 ceph-mon[33359]: pgmap v1749: 188 pgs: 188 active+clean; 858 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-11T21:26:30.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:30 smithi129 ceph-mon[30184]: pgmap v1750: 188 pgs: 188 active+clean; 858 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-11T21:26:30.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:30 smithi129 ceph-mon[30184]: pool '4bf093d5-8685-43fb-a522-cbae7e0b1dcf' is full (reached quota's max_objects: 10) 2024-03-11T21:26:30.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:30 smithi129 ceph-mon[30184]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-11T21:26:30.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:30 smithi129 ceph-mon[30184]: osdmap e786: 8 total, 8 up, 8 in 2024-03-11T21:26:30.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:30 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:26:30.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:30 smithi129 ceph-mon[42066]: pgmap v1750: 188 pgs: 188 active+clean; 858 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-11T21:26:30.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:30 smithi129 ceph-mon[42066]: pool '4bf093d5-8685-43fb-a522-cbae7e0b1dcf' is full (reached quota's max_objects: 10) 2024-03-11T21:26:30.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:30 smithi129 ceph-mon[42066]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-11T21:26:30.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:30 smithi129 ceph-mon[42066]: osdmap e786: 8 total, 8 up, 8 in 2024-03-11T21:26:30.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:30 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:26:30.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:30 smithi157 ceph-mon[33359]: pgmap v1750: 188 pgs: 188 active+clean; 858 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-11T21:26:30.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:30 smithi157 ceph-mon[33359]: pool '4bf093d5-8685-43fb-a522-cbae7e0b1dcf' is full (reached quota's max_objects: 10) 2024-03-11T21:26:30.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:30 smithi157 ceph-mon[33359]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-11T21:26:30.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:30 smithi157 ceph-mon[33359]: osdmap e786: 8 total, 8 up, 8 in 2024-03-11T21:26:30.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:30 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:26:31.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:31 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:26:31.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:31 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:26:31.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:31 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:26:31.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:31 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:26:31.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:31 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:26:31.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:31 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:26:31.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:31 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:26:31.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:31 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:26:31.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:31 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:26:32.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:32 smithi129 ceph-mon[30184]: pgmap v1752: 188 pgs: 188 active+clean; 858 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-11T21:26:32.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:32 smithi129 ceph-mon[42066]: pgmap v1752: 188 pgs: 188 active+clean; 858 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-11T21:26:32.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:32 smithi157 ceph-mon[33359]: pgmap v1752: 188 pgs: 188 active+clean; 858 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-11T21:26:34.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:34 smithi129 ceph-mon[30184]: pgmap v1753: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 8.4 KiB/s wr, 1 op/s 2024-03-11T21:26:34.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:34 smithi129 ceph-mon[42066]: pgmap v1753: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 8.4 KiB/s wr, 1 op/s 2024-03-11T21:26:34.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:34 smithi157 ceph-mon[33359]: pgmap v1753: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 8.4 KiB/s wr, 1 op/s 2024-03-11T21:26:35.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:26:35 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:26:36.465 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:26:36 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:26:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:26:36.466 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:35 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:26:36.466 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:35 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:36.466 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:35 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:36.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:35 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:26:36.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:35 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:36.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:35 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:36.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:35 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' 2024-03-11T21:26:36.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:35 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:36.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:35 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:37.465 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:36 smithi129 ceph-mon[30184]: pgmap v1754: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 8.4 KiB/s wr, 1 op/s 2024-03-11T21:26:37.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:37 smithi129 ceph-mon[42066]: pgmap v1754: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 8.4 KiB/s wr, 1 op/s 2024-03-11T21:26:37.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:36 smithi157 ceph-mon[33359]: pgmap v1754: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 8.4 KiB/s wr, 1 op/s 2024-03-11T21:26:39.465 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:39 smithi129 ceph-mon[30184]: pgmap v1755: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:39.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:39 smithi129 ceph-mon[42066]: pgmap v1755: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:39.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:39 smithi157 ceph-mon[33359]: pgmap v1755: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:41.465 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:41 smithi129 ceph-mon[30184]: pgmap v1756: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:41.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:41 smithi129 ceph-mon[42066]: pgmap v1756: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:41.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:41 smithi157 ceph-mon[33359]: pgmap v1756: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:43.466 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:43 smithi129 ceph-mon[30184]: pgmap v1757: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-03-11T21:26:43.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:43 smithi129 ceph-mon[42066]: pgmap v1757: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-03-11T21:26:43.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:43 smithi157 ceph-mon[33359]: pgmap v1757: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-03-11T21:26:45.275 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:45 smithi157 ceph-mon[33359]: pgmap v1758: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:45.276 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:26:45 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:26:45.465 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:45 smithi129 ceph-mon[30184]: pgmap v1758: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:45.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:45 smithi129 ceph-mon[42066]: pgmap v1758: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:46.465 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:26:46 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:26:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:26:47.465 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:47 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:47.465 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:47 smithi129 ceph-mon[30184]: pgmap v1759: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:47.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:47 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:47.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:47 smithi129 ceph-mon[42066]: pgmap v1759: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:47.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:47 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:47.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:47 smithi157 ceph-mon[33359]: pgmap v1759: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:49.465 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:49 smithi129 ceph-mon[42066]: pgmap v1760: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:49.466 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:49 smithi129 ceph-mon[30184]: pgmap v1760: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:49.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:49 smithi157 ceph-mon[33359]: pgmap v1760: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:50.465 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:50 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:50.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:50 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:50.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:50 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:51.465 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:51 smithi129 ceph-mon[30184]: pgmap v1761: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:51.465 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:51 smithi129 ceph-mon[42066]: pgmap v1761: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:51.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:51 smithi157 ceph-mon[33359]: pgmap v1761: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:51.972 INFO:tasks.workunit.client.0.smithi129.stderr:+ rados -p ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 put threemore /etc/passwd 2024-03-11T21:26:52.014 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph osd pool set-quota ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 max_bytes 0 2024-03-11T21:26:52.104 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.118+0000 7f73d7d37640 1 -- 172.21.15.129:0/1071358292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d0101ee0 msgr2=0x7f73d0105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:52.104 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.118+0000 7f73d7d37640 1 --2- 172.21.15.129:0/1071358292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d0101ee0 0x7f73d0105f20 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 crypto rx=0x7f73c0009860 tx=0x7f73c002bb70 comp rx=0 tx=0).stop 2024-03-11T21:26:52.104 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.118+0000 7f73d7d37640 1 -- 172.21.15.129:0/1071358292 shutdown_connections 2024-03-11T21:26:52.104 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.118+0000 7f73d7d37640 1 --2- 172.21.15.129:0/1071358292 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73d0106600 0x7f73d010e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:52.104 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.118+0000 7f73d7d37640 1 --2- 172.21.15.129:0/1071358292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d0101ee0 0x7f73d0105f20 unknown :-1 s=CLOSED pgs=3041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:52.104 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.118+0000 7f73d7d37640 1 --2- 172.21.15.129:0/1071358292 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f73d0101530 0x7f73d0101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:52.104 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.118+0000 7f73d7d37640 1 -- 172.21.15.129:0/1071358292 >> 172.21.15.129:0/1071358292 conn(0x7f73d00fcdf0 msgr2=0x7f73d00ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:52.104 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.118+0000 7f73d7d37640 1 -- 172.21.15.129:0/1071358292 shutdown_connections 2024-03-11T21:26:52.104 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.118+0000 7f73d7d37640 1 -- 172.21.15.129:0/1071358292 wait complete. 2024-03-11T21:26:52.104 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.118+0000 7f73d7d37640 1 Processor -- start 2024-03-11T21:26:52.105 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.118+0000 7f73d7d37640 1 -- start start 2024-03-11T21:26:52.105 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d7d37640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f73d0101530 0x7f73d0197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:52.105 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d7d37640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73d0106600 0x7f73d01976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:52.105 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d7d37640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d019ba20 0x7f73d019bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:52.105 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d7d37640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d0110b00 con 0x7f73d019ba20 2024-03-11T21:26:52.105 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d7d37640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d0110980 con 0x7f73d0106600 2024-03-11T21:26:52.105 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d7d37640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d0110c80 con 0x7f73d0101530 2024-03-11T21:26:52.105 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d5ad3640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f73d0101530 0x7f73d0197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:52.105 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d62d4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d019ba20 0x7f73d019bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:52.105 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d62d4640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d019ba20 0x7f73d019bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57954/0 (socket says 172.21.15.129:57954) 2024-03-11T21:26:52.105 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d5ad3640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f73d0101530 0x7f73d0197160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:57494/0 (socket says 172.21.15.129:57494) 2024-03-11T21:26:52.105 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d62d4640 1 -- 172.21.15.129:0/3559186071 learned_addr learned my addr 172.21.15.129:0/3559186071 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T21:26:52.106 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d52d2640 1 --2- 172.21.15.129:0/3559186071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73d0106600 0x7f73d01976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:52.106 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d62d4640 1 -- 172.21.15.129:0/3559186071 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f73d0101530 msgr2=0x7f73d0197160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:52.106 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d62d4640 1 --2- 172.21.15.129:0/3559186071 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f73d0101530 0x7f73d0197160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:52.106 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d62d4640 1 -- 172.21.15.129:0/3559186071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73d0106600 msgr2=0x7f73d01976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:52.106 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d62d4640 1 --2- 172.21.15.129:0/3559186071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73d0106600 0x7f73d01976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:52.106 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d62d4640 1 -- 172.21.15.129:0/3559186071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d019c510 con 0x7f73d019ba20 2024-03-11T21:26:52.106 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d52d2640 1 --2- 172.21.15.129:0/3559186071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73d0106600 0x7f73d01976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:26:52.106 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d5ad3640 1 --2- 172.21.15.129:0/3559186071 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f73d0101530 0x7f73d0197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:26:52.106 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d62d4640 1 --2- 172.21.15.129:0/3559186071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d019ba20 0x7f73d019bed0 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7f73c800eaa0 tx=0x7f73c800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:52.106 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73c6ffd640 1 -- 172.21.15.129:0/3559186071 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c800cdb0 con 0x7f73d019ba20 2024-03-11T21:26:52.106 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73c6ffd640 1 -- 172.21.15.129:0/3559186071 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f73c8005020 con 0x7f73d019ba20 2024-03-11T21:26:52.106 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d7d37640 1 -- 172.21.15.129:0/3559186071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d019c800 con 0x7f73d019ba20 2024-03-11T21:26:52.106 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73c6ffd640 1 -- 172.21.15.129:0/3559186071 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c80116d0 con 0x7f73d019ba20 2024-03-11T21:26:52.107 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.119+0000 7f73d7d37640 1 -- 172.21.15.129:0/3559186071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d01a3f70 con 0x7f73d019ba20 2024-03-11T21:26:52.107 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.120+0000 7f73d7d37640 1 -- 172.21.15.129:0/3559186071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d0068620 con 0x7f73d019ba20 2024-03-11T21:26:52.107 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.121+0000 7f73c6ffd640 1 -- 172.21.15.129:0/3559186071 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f73c8004a40 con 0x7f73d019ba20 2024-03-11T21:26:52.107 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.121+0000 7f73c6ffd640 1 --2- 172.21.15.129:0/3559186071 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f73bc06ddf0 0x7f73bc0702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:52.108 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.121+0000 7f73c6ffd640 1 -- 172.21.15.129:0/3559186071 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(786..786 src has 254..786) v4 ==== 9407+0+0 (secure 0 0 0) 0x7f73c808eb60 con 0x7f73d019ba20 2024-03-11T21:26:52.108 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.121+0000 7f73d5ad3640 1 --2- 172.21.15.129:0/3559186071 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f73bc06ddf0 0x7f73bc0702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:52.108 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.122+0000 7f73c6ffd640 1 -- 172.21.15.129:0/3559186071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=787}) v3 -- 0x7f73bc07a000 con 0x7f73d019ba20 2024-03-11T21:26:52.108 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.122+0000 7f73d5ad3640 1 --2- 172.21.15.129:0/3559186071 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f73bc06ddf0 0x7f73bc0702b0 secure :-1 s=READY pgs=4057 cs=0 l=1 rev1=1 crypto rx=0x7f73cc0096c0 tx=0x7f73cc009260 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:52.109 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.126+0000 7f73c6ffd640 1 -- 172.21.15.129:0/3559186071 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f73c8016030 con 0x7f73d019ba20 2024-03-11T21:26:52.285 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:52.302+0000 7f73d7d37640 1 -- 172.21.15.129:0/3559186071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f73d0101ee0 con 0x7f73d019ba20 2024-03-11T21:26:53.005 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:53.022+0000 7f73c6ffd640 1 -- 172.21.15.129:0/3559186071 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 v787) v1 ==== 217+0+0 (secure 0 0 0) 0x7f73c805f2e0 con 0x7f73d019ba20 2024-03-11T21:26:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:53.025+0000 7f73c6ffd640 1 -- 172.21.15.129:0/3559186071 <== mon.0 v2:172.21.15.129:3300/0 8 ==== osd_map(787..787 src has 254..787) v4 ==== 628+0+0 (secure 0 0 0) 0x7f73c80592e0 con 0x7f73d019ba20 2024-03-11T21:26:53.008 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:53.025+0000 7f73c6ffd640 1 -- 172.21.15.129:0/3559186071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=788}) v3 -- 0x7f73bc07afa0 con 0x7f73d019ba20 2024-03-11T21:26:53.120 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:53.137+0000 7f73d7d37640 1 -- 172.21.15.129:0/3559186071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f73d01a4200 con 0x7f73d019ba20 2024-03-11T21:26:53.465 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:53 smithi129 ceph-mon[42066]: pgmap v1762: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:53.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:53 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3559186071' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-11T21:26:53.466 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:53 smithi129 ceph-mon[30184]: pgmap v1762: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:53.466 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:53 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3559186071' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-11T21:26:53.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:53 smithi157 ceph-mon[33359]: pgmap v1762: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:53.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:53 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3559186071' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-11T21:26:54.024 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.041+0000 7f73c6ffd640 1 -- 172.21.15.129:0/3559186071 <== mon.0 v2:172.21.15.129:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 v788) v1 ==== 217+0+0 (secure 0 0 0) 0x7f73c80638a0 con 0x7f73d019ba20 2024-03-11T21:26:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:set-quota max_bytes = 0 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 2024-03-11T21:26:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.041+0000 7f73c6ffd640 1 -- 172.21.15.129:0/3559186071 <== mon.0 v2:172.21.15.129:3300/0 10 ==== osd_map(788..788 src has 254..788) v4 ==== 628+0+0 (secure 0 0 0) 0x7f73c80592e0 con 0x7f73d019ba20 2024-03-11T21:26:54.025 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.041+0000 7f73c6ffd640 1 -- 172.21.15.129:0/3559186071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=789}) v3 -- 0x7f73bc07b1b0 con 0x7f73d019ba20 2024-03-11T21:26:54.029 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.046+0000 7f73d7d37640 1 -- 172.21.15.129:0/3559186071 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f73bc06ddf0 msgr2=0x7f73bc0702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:54.029 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.046+0000 7f73d7d37640 1 --2- 172.21.15.129:0/3559186071 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f73bc06ddf0 0x7f73bc0702b0 secure :-1 s=READY pgs=4057 cs=0 l=1 rev1=1 crypto rx=0x7f73cc0096c0 tx=0x7f73cc009260 comp rx=0 tx=0).stop 2024-03-11T21:26:54.029 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.046+0000 7f73d7d37640 1 -- 172.21.15.129:0/3559186071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d019ba20 msgr2=0x7f73d019bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:54.029 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.046+0000 7f73d7d37640 1 --2- 172.21.15.129:0/3559186071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d019ba20 0x7f73d019bed0 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7f73c800eaa0 tx=0x7f73c800ef60 comp rx=0 tx=0).stop 2024-03-11T21:26:54.029 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.046+0000 7f73d7d37640 1 -- 172.21.15.129:0/3559186071 shutdown_connections 2024-03-11T21:26:54.029 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.046+0000 7f73d7d37640 1 --2- 172.21.15.129:0/3559186071 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f73bc06ddf0 0x7f73bc0702b0 unknown :-1 s=CLOSED pgs=4057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:54.029 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.046+0000 7f73d7d37640 1 --2- 172.21.15.129:0/3559186071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73d019ba20 0x7f73d019bed0 unknown :-1 s=CLOSED pgs=3042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:54.029 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.046+0000 7f73d7d37640 1 --2- 172.21.15.129:0/3559186071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f73d0106600 0x7f73d01976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:54.029 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.046+0000 7f73d7d37640 1 --2- 172.21.15.129:0/3559186071 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f73d0101530 0x7f73d0197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:54.029 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.046+0000 7f73d7d37640 1 -- 172.21.15.129:0/3559186071 >> 172.21.15.129:0/3559186071 conn(0x7f73d00fcdf0 msgr2=0x7f73d00fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:54.030 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.048+0000 7f73d7d37640 1 -- 172.21.15.129:0/3559186071 shutdown_connections 2024-03-11T21:26:54.031 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.048+0000 7f73d7d37640 1 -- 172.21.15.129:0/3559186071 wait complete. 2024-03-11T21:26:54.043 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph osd pool set-quota ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 max_objects 0 2024-03-11T21:26:54.132 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.147+0000 7f3f96883640 1 -- 172.21.15.129:0/2939974853 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3f90100440 msgr2=0x7f3f901008c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:54.132 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.147+0000 7f3f96883640 1 --2- 172.21.15.129:0/2939974853 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3f90100440 0x7f3f901008c0 secure :-1 s=READY pgs=3108 cs=0 l=1 rev1=1 crypto rx=0x7f3f84009930 tx=0x7f3f8402bc30 comp rx=0 tx=0).stop 2024-03-11T21:26:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.147+0000 7f3f96883640 1 -- 172.21.15.129:0/2939974853 shutdown_connections 2024-03-11T21:26:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.147+0000 7f3f96883640 1 --2- 172.21.15.129:0/2939974853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f90100e00 0x7f3f9010cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.147+0000 7f3f96883640 1 --2- 172.21.15.129:0/2939974853 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3f90100440 0x7f3f901008c0 unknown :-1 s=CLOSED pgs=3108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.147+0000 7f3f96883640 1 --2- 172.21.15.129:0/2939974853 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f90106380 0x7f3f90106760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.147+0000 7f3f96883640 1 -- 172.21.15.129:0/2939974853 >> 172.21.15.129:0/2939974853 conn(0x7f3f90076fb0 msgr2=0x7f3f900773c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.147+0000 7f3f96883640 1 -- 172.21.15.129:0/2939974853 shutdown_connections 2024-03-11T21:26:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.147+0000 7f3f96883640 1 -- 172.21.15.129:0/2939974853 wait complete. 2024-03-11T21:26:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.147+0000 7f3f96883640 1 Processor -- start 2024-03-11T21:26:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.147+0000 7f3f96883640 1 -- start start 2024-03-11T21:26:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f96883640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f90100440 0x7f3f90196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:54.133 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f96883640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f90100e00 0x7f3f90196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f96883640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3f90106380 0x7f3f9019a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f96883640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9010fa60 con 0x7f3f90100e00 2024-03-11T21:26:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f96883640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9010f8e0 con 0x7f3f90100440 2024-03-11T21:26:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f96883640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9010fbe0 con 0x7f3f90106380 2024-03-11T21:26:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f8f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f90100e00 0x7f3f90196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f94e20640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3f90106380 0x7f3f9019a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f8f7fe640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f90100e00 0x7f3f90196570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57968/0 (socket says 172.21.15.129:57968) 2024-03-11T21:26:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f8f7fe640 1 -- 172.21.15.129:0/3152261060 learned_addr learned my addr 172.21.15.129:0/3152261060 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T21:26:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f8ffff640 1 --2- 172.21.15.129:0/3152261060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f90100440 0x7f3f90196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f8f7fe640 1 -- 172.21.15.129:0/3152261060 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3f90106380 msgr2=0x7f3f9019a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f8f7fe640 1 --2- 172.21.15.129:0/3152261060 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3f90106380 0x7f3f9019a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f8f7fe640 1 -- 172.21.15.129:0/3152261060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f90100440 msgr2=0x7f3f90196030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:54.134 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f8f7fe640 1 --2- 172.21.15.129:0/3152261060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f90100440 0x7f3f90196030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f8f7fe640 1 -- 172.21.15.129:0/3152261060 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9019b020 con 0x7f3f90100e00 2024-03-11T21:26:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f94e20640 1 --2- 172.21.15.129:0/3152261060 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3f90106380 0x7f3f9019a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:26:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f8f7fe640 1 --2- 172.21.15.129:0/3152261060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f90100e00 0x7f3f90196570 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7f3f84000c00 tx=0x7f3f84004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f8d7fa640 1 -- 172.21.15.129:0/3152261060 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f84043020 con 0x7f3f90100e00 2024-03-11T21:26:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f8d7fa640 1 -- 172.21.15.129:0/3152261060 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f3f84004a30 con 0x7f3f90100e00 2024-03-11T21:26:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f96883640 1 -- 172.21.15.129:0/3152261060 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f9019b2b0 con 0x7f3f90100e00 2024-03-11T21:26:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.148+0000 7f3f8d7fa640 1 -- 172.21.15.129:0/3152261060 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f84003860 con 0x7f3f90100e00 2024-03-11T21:26:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.149+0000 7f3f96883640 1 -- 172.21.15.129:0/3152261060 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f90069a90 con 0x7f3f90100e00 2024-03-11T21:26:54.135 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.149+0000 7f3f96883640 1 -- 172.21.15.129:0/3152261060 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f901018e0 con 0x7f3f90100e00 2024-03-11T21:26:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.150+0000 7f3f8d7fa640 1 -- 172.21.15.129:0/3152261060 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f3f84046020 con 0x7f3f90100e00 2024-03-11T21:26:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.150+0000 7f3f8d7fa640 1 --2- 172.21.15.129:0/3152261060 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f3f7806ddf0 0x7f3f780702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:26:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.150+0000 7f3f8ffff640 1 --2- 172.21.15.129:0/3152261060 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f3f7806ddf0 0x7f3f780702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:26:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.150+0000 7f3f8d7fa640 1 -- 172.21.15.129:0/3152261060 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(788..788 src has 254..788) v4 ==== 9407+0+0 (secure 0 0 0) 0x7f3f840b0a70 con 0x7f3f90100e00 2024-03-11T21:26:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.151+0000 7f3f8d7fa640 1 -- 172.21.15.129:0/3152261060 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=789}) v3 -- 0x7f3f7807a000 con 0x7f3f90100e00 2024-03-11T21:26:54.136 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.151+0000 7f3f8ffff640 1 --2- 172.21.15.129:0/3152261060 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f3f7806ddf0 0x7f3f780702b0 secure :-1 s=READY pgs=4058 cs=0 l=1 rev1=1 crypto rx=0x7f3f800095f0 tx=0x7f3f80005f90 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:26:54.138 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.155+0000 7f3f8d7fa640 1 -- 172.21.15.129:0/3152261060 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f3f8404d020 con 0x7f3f90100e00 2024-03-11T21:26:54.314 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:54.331+0000 7f3f96883640 1 -- 172.21.15.129:0/3152261060 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f3f90197380 con 0x7f3f90100e00 2024-03-11T21:26:54.465 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3559186071' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]': finished 2024-03-11T21:26:54.465 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:54 smithi129 ceph-mon[30184]: osdmap e787: 8 total, 8 up, 8 in 2024-03-11T21:26:54.466 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:54 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3559186071' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-11T21:26:54.466 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:54 smithi129 ceph-mon[30184]: pgmap v1764: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:54.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3559186071' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]': finished 2024-03-11T21:26:54.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:54 smithi129 ceph-mon[42066]: osdmap e787: 8 total, 8 up, 8 in 2024-03-11T21:26:54.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:54 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3559186071' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-11T21:26:54.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:54 smithi129 ceph-mon[42066]: pgmap v1764: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:54.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3559186071' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]': finished 2024-03-11T21:26:54.474 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:54 smithi157 ceph-mon[33359]: osdmap e787: 8 total, 8 up, 8 in 2024-03-11T21:26:54.474 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:54 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3559186071' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-11T21:26:54.474 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:54 smithi157 ceph-mon[33359]: pgmap v1764: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:55.029 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:55.046+0000 7f3f8d7fa640 1 -- 172.21.15.129:0/3152261060 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 v789) v1 ==== 221+0+0 (secure 0 0 0) 0x7f3f84005430 con 0x7f3f90100e00 2024-03-11T21:26:55.030 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:55.046+0000 7f3f8d7fa640 1 -- 172.21.15.129:0/3152261060 <== mon.0 v2:172.21.15.129:3300/0 8 ==== osd_map(789..789 src has 254..789) v4 ==== 628+0+0 (secure 0 0 0) 0x7f3f8407b1f0 con 0x7f3f90100e00 2024-03-11T21:26:55.030 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:55.047+0000 7f3f8d7fa640 1 -- 172.21.15.129:0/3152261060 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=790}) v3 -- 0x7f3f7807afa0 con 0x7f3f90100e00 2024-03-11T21:26:55.139 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:55.156+0000 7f3f96883640 1 -- 172.21.15.129:0/3152261060 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f3f90069db0 con 0x7f3f90100e00 2024-03-11T21:26:55.465 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3559186071' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]': finished 2024-03-11T21:26:55.466 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:55 smithi129 ceph-mon[30184]: osdmap e788: 8 total, 8 up, 8 in 2024-03-11T21:26:55.466 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:55 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3152261060' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:26:55.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3559186071' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]': finished 2024-03-11T21:26:55.467 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:55 smithi129 ceph-mon[42066]: osdmap e788: 8 total, 8 up, 8 in 2024-03-11T21:26:55.467 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:55 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3152261060' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:26:55.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:26:55 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:26:55.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3559186071' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_bytes", "val": "0"}]': finished 2024-03-11T21:26:55.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:55 smithi157 ceph-mon[33359]: osdmap e788: 8 total, 8 up, 8 in 2024-03-11T21:26:55.474 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:55 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3152261060' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:26:56.037 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:56.054+0000 7f3f8d7fa640 1 -- 172.21.15.129:0/3152261060 <== mon.0 v2:172.21.15.129:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 v790) v1 ==== 221+0+0 (secure 0 0 0) 0x7f3f840811f0 con 0x7f3f90100e00 2024-03-11T21:26:56.038 INFO:tasks.workunit.client.0.smithi129.stderr:set-quota max_objects = 0 for pool ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 2024-03-11T21:26:56.040 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:56.057+0000 7f3f8d7fa640 1 -- 172.21.15.129:0/3152261060 <== mon.0 v2:172.21.15.129:3300/0 10 ==== osd_map(790..790 src has 254..790) v4 ==== 628+0+0 (secure 0 0 0) 0x7f3f840ae1c0 con 0x7f3f90100e00 2024-03-11T21:26:56.040 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:56.057+0000 7f3f8d7fa640 1 -- 172.21.15.129:0/3152261060 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=791}) v3 -- 0x7f3f7807b090 con 0x7f3f90100e00 2024-03-11T21:26:56.041 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:56.058+0000 7f3f96883640 1 -- 172.21.15.129:0/3152261060 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f3f7806ddf0 msgr2=0x7f3f780702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:56.041 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:56.058+0000 7f3f96883640 1 --2- 172.21.15.129:0/3152261060 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f3f7806ddf0 0x7f3f780702b0 secure :-1 s=READY pgs=4058 cs=0 l=1 rev1=1 crypto rx=0x7f3f800095f0 tx=0x7f3f80005f90 comp rx=0 tx=0).stop 2024-03-11T21:26:56.041 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:56.058+0000 7f3f96883640 1 -- 172.21.15.129:0/3152261060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f90100e00 msgr2=0x7f3f90196570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:26:56.041 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:56.058+0000 7f3f96883640 1 --2- 172.21.15.129:0/3152261060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f90100e00 0x7f3f90196570 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7f3f84000c00 tx=0x7f3f84004270 comp rx=0 tx=0).stop 2024-03-11T21:26:56.041 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:56.058+0000 7f3f96883640 1 -- 172.21.15.129:0/3152261060 shutdown_connections 2024-03-11T21:26:56.041 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:56.058+0000 7f3f96883640 1 --2- 172.21.15.129:0/3152261060 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f3f7806ddf0 0x7f3f780702b0 unknown :-1 s=CLOSED pgs=4058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:56.042 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:56.058+0000 7f3f96883640 1 --2- 172.21.15.129:0/3152261060 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f3f90106380 0x7f3f9019a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:56.042 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:56.058+0000 7f3f96883640 1 --2- 172.21.15.129:0/3152261060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f90100e00 0x7f3f90196570 unknown :-1 s=CLOSED pgs=3043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:56.042 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:56.058+0000 7f3f96883640 1 --2- 172.21.15.129:0/3152261060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f90100440 0x7f3f90196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:26:56.042 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:56.058+0000 7f3f96883640 1 -- 172.21.15.129:0/3152261060 >> 172.21.15.129:0/3152261060 conn(0x7f3f90076fb0 msgr2=0x7f3f90103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:26:56.042 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:56.058+0000 7f3f96883640 1 -- 172.21.15.129:0/3152261060 shutdown_connections 2024-03-11T21:26:56.042 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:26:56.058+0000 7f3f96883640 1 -- 172.21.15.129:0/3152261060 wait complete. 2024-03-11T21:26:56.055 INFO:tasks.workunit.client.0.smithi129.stderr:+ sleep 30 2024-03-11T21:26:56.367 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:26:56 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:26:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:26:56.367 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:56 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:56.367 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3152261060' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]': finished 2024-03-11T21:26:56.367 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:56 smithi129 ceph-mon[30184]: osdmap e789: 8 total, 8 up, 8 in 2024-03-11T21:26:56.367 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:56 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3152261060' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:26:56.367 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:56 smithi129 ceph-mon[30184]: pgmap v1767: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:56.368 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:56 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:56.368 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3152261060' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]': finished 2024-03-11T21:26:56.368 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:56 smithi129 ceph-mon[42066]: osdmap e789: 8 total, 8 up, 8 in 2024-03-11T21:26:56.368 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:56 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3152261060' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:26:56.368 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:56 smithi129 ceph-mon[42066]: pgmap v1767: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:56.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:56 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:56.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3152261060' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]': finished 2024-03-11T21:26:56.474 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:56 smithi157 ceph-mon[33359]: osdmap e789: 8 total, 8 up, 8 in 2024-03-11T21:26:56.474 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:56 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3152261060' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]: dispatch 2024-03-11T21:26:56.474 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:56 smithi157 ceph-mon[33359]: pgmap v1767: 188 pgs: 188 active+clean; 858 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:57.465 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:57 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/3152261060' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]': finished 2024-03-11T21:26:57.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:57 smithi129 ceph-mon[42066]: osdmap e790: 8 total, 8 up, 8 in 2024-03-11T21:26:57.466 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:57 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/3152261060' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]': finished 2024-03-11T21:26:57.466 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:57 smithi129 ceph-mon[30184]: osdmap e790: 8 total, 8 up, 8 in 2024-03-11T21:26:57.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:57 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/3152261060' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "field": "max_objects", "val": "0"}]': finished 2024-03-11T21:26:57.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:57 smithi157 ceph-mon[33359]: osdmap e790: 8 total, 8 up, 8 in 2024-03-11T21:26:58.465 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:26:58 smithi129 ceph-mon[30184]: pgmap v1769: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.7 KiB/s wr, 1 op/s 2024-03-11T21:26:58.466 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:26:58 smithi129 ceph-mon[42066]: pgmap v1769: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.7 KiB/s wr, 1 op/s 2024-03-11T21:26:58.473 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:26:58 smithi157 ceph-mon[33359]: pgmap v1769: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.7 KiB/s wr, 1 op/s 2024-03-11T21:27:00.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:00 smithi129 ceph-mon[30184]: pgmap v1770: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-11T21:27:00.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:00 smithi129 ceph-mon[42066]: pgmap v1770: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-11T21:27:00.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:00 smithi157 ceph-mon[33359]: pgmap v1770: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-11T21:27:02.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:02 smithi129 ceph-mon[30184]: pgmap v1771: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 689 B/s rd, 1.6 KiB/s wr, 0 op/s 2024-03-11T21:27:02.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:02 smithi129 ceph-mon[42066]: pgmap v1771: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 689 B/s rd, 1.6 KiB/s wr, 0 op/s 2024-03-11T21:27:02.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:02 smithi157 ceph-mon[33359]: pgmap v1771: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 689 B/s rd, 1.6 KiB/s wr, 0 op/s 2024-03-11T21:27:04.965 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:04 smithi129 ceph-mon[42066]: pgmap v1772: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 1 op/s 2024-03-11T21:27:04.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:04 smithi129 ceph-mon[30184]: pgmap v1772: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 1 op/s 2024-03-11T21:27:04.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:04 smithi157 ceph-mon[33359]: pgmap v1772: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 1 op/s 2024-03-11T21:27:05.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:27:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:27:05.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:05 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:05.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:05 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:05.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:05 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:06.465 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:27:06 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:27:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:27:06.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:06 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:06.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:06 smithi129 ceph-mon[30184]: pgmap v1773: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-11T21:27:06.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:06 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:06.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:06 smithi129 ceph-mon[42066]: pgmap v1773: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-11T21:27:06.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:06 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:06.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:06 smithi157 ceph-mon[33359]: pgmap v1773: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-11T21:27:08.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:08 smithi129 ceph-mon[30184]: pgmap v1774: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.1 KiB/s wr, 1 op/s 2024-03-11T21:27:08.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:08 smithi129 ceph-mon[42066]: pgmap v1774: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.1 KiB/s wr, 1 op/s 2024-03-11T21:27:08.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:08 smithi157 ceph-mon[33359]: pgmap v1774: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.1 KiB/s wr, 1 op/s 2024-03-11T21:27:10.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:10 smithi129 ceph-mon[30184]: pgmap v1775: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:10.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:10 smithi129 ceph-mon[42066]: pgmap v1775: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:10.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:10 smithi157 ceph-mon[33359]: pgmap v1775: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:12.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:12 smithi129 ceph-mon[30184]: pgmap v1776: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:12.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:12 smithi129 ceph-mon[42066]: pgmap v1776: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:12.972 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:12 smithi157 ceph-mon[33359]: pgmap v1776: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:14.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:14 smithi129 ceph-mon[30184]: pgmap v1777: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:14.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:14 smithi129 ceph-mon[42066]: pgmap v1777: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:14.972 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:14 smithi157 ceph-mon[33359]: pgmap v1777: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:15.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:27:15 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:27:16.465 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:27:16 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:27:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:27:16.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:16 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:16.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:16 smithi129 ceph-mon[30184]: pgmap v1778: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:27:16.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:16 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:16.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:16 smithi129 ceph-mon[42066]: pgmap v1778: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:27:16.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:16 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:16.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:16 smithi157 ceph-mon[33359]: pgmap v1778: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:27:18.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:18 smithi129 ceph-mon[30184]: pgmap v1779: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:18.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:18 smithi129 ceph-mon[42066]: pgmap v1779: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:18.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:18 smithi157 ceph-mon[33359]: pgmap v1779: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:20.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:20 smithi129 ceph-mon[30184]: pgmap v1780: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:27:20.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:20 smithi129 ceph-mon[30184]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:20.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:20 smithi129 ceph-mon[42066]: pgmap v1780: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:27:20.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:20 smithi129 ceph-mon[42066]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:20.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:20 smithi157 ceph-mon[33359]: pgmap v1780: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-11T21:27:20.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:20 smithi157 ceph-mon[33359]: from='mgr.24497 172.21.15.129:0/1430063547' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:22.965 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:22 smithi129 ceph-mon[42066]: pgmap v1781: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:22.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:22 smithi129 ceph-mon[30184]: pgmap v1781: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:22.972 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:22 smithi157 ceph-mon[33359]: pgmap v1781: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:24.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:24 smithi129 ceph-mon[30184]: pgmap v1782: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:24.965 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:24 smithi129 ceph-mon[42066]: pgmap v1782: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:24.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:24 smithi157 ceph-mon[33359]: pgmap v1782: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:25.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:27:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug there is no tcmu-runner data available 2024-03-11T21:27:26.056 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph osd pool delete ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 ff0f5cc6-20a7-4630-8c16-3bfb73edfea7 --yes-i-really-really-mean-it 2024-03-11T21:27:26.151 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.165+0000 7f5bfe481640 1 -- 172.21.15.129:0/67105081 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf8102560 msgr2=0x7f5bf8102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:26.151 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.165+0000 7f5bfe481640 1 --2- 172.21.15.129:0/67105081 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf8102560 0x7f5bf8102940 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7f5be8009900 tx=0x7f5be802bcd0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.151 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.165+0000 7f5bfe481640 1 -- 172.21.15.129:0/67105081 shutdown_connections 2024-03-11T21:27:26.151 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.165+0000 7f5bfe481640 1 --2- 172.21.15.129:0/67105081 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5bf810b930 0x7f5bf810dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.151 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.165+0000 7f5bfe481640 1 --2- 172.21.15.129:0/67105081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bf8102e80 0x7f5bf810b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.151 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.165+0000 7f5bfe481640 1 --2- 172.21.15.129:0/67105081 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf8102560 0x7f5bf8102940 unknown :-1 s=CLOSED pgs=3044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.151 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.165+0000 7f5bfe481640 1 -- 172.21.15.129:0/67105081 >> 172.21.15.129:0/67105081 conn(0x7f5bf80772e0 msgr2=0x7f5bf80776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:27:26.152 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.166+0000 7f5bfe481640 1 -- 172.21.15.129:0/67105081 shutdown_connections 2024-03-11T21:27:26.152 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.166+0000 7f5bfe481640 1 -- 172.21.15.129:0/67105081 wait complete. 2024-03-11T21:27:26.152 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.166+0000 7f5bfe481640 1 Processor -- start 2024-03-11T21:27:26.152 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.166+0000 7f5bfe481640 1 -- start start 2024-03-11T21:27:26.152 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.166+0000 7f5bfe481640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bf8102560 0x7f5bf8197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:26.152 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.166+0000 7f5bfe481640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5bf8102e80 0x7f5bf8197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:26.152 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.166+0000 7f5bfe481640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf810b930 0x7f5bf819b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:26.152 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.166+0000 7f5bfe481640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf8110b10 con 0x7f5bf810b930 2024-03-11T21:27:26.152 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bfe481640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf8110990 con 0x7f5bf8102560 2024-03-11T21:27:26.152 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bfe481640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf8110c90 con 0x7f5bf8102e80 2024-03-11T21:27:26.152 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bfca1e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf810b930 0x7f5bf819b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:26.152 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bf77fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5bf8102e80 0x7f5bf8197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:26.152 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bfca1e640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf810b930 0x7f5bf819b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52122/0 (socket says 172.21.15.129:52122) 2024-03-11T21:27:26.153 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bf77fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5bf8102e80 0x7f5bf8197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3301/0 says I am v2:172.21.15.129:54986/0 (socket says 172.21.15.129:54986) 2024-03-11T21:27:26.153 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bfca1e640 1 -- 172.21.15.129:0/1131851907 learned_addr learned my addr 172.21.15.129:0/1131851907 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T21:27:26.153 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bf7fff640 1 --2- 172.21.15.129:0/1131851907 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bf8102560 0x7f5bf8197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:26.153 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bfca1e640 1 -- 172.21.15.129:0/1131851907 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5bf8102e80 msgr2=0x7f5bf8197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:26.153 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bfca1e640 1 --2- 172.21.15.129:0/1131851907 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5bf8102e80 0x7f5bf8197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.153 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bfca1e640 1 -- 172.21.15.129:0/1131851907 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bf8102560 msgr2=0x7f5bf8197130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:26.153 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bfca1e640 1 --2- 172.21.15.129:0/1131851907 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bf8102560 0x7f5bf8197130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.153 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bfca1e640 1 -- 172.21.15.129:0/1131851907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf819c120 con 0x7f5bf810b930 2024-03-11T21:27:26.153 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bf77fe640 1 --2- 172.21.15.129:0/1131851907 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5bf8102e80 0x7f5bf8197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:27:26.153 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bfca1e640 1 --2- 172.21.15.129:0/1131851907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf810b930 0x7f5bf819b9f0 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0009c90 tx=0x7f5bf000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:26.153 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bf57fa640 1 -- 172.21.15.129:0/1131851907 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0003b70 con 0x7f5bf810b930 2024-03-11T21:27:26.153 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bf57fa640 1 -- 172.21.15.129:0/1131851907 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5bf0003d10 con 0x7f5bf810b930 2024-03-11T21:27:26.153 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bfe481640 1 -- 172.21.15.129:0/1131851907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf819c410 con 0x7f5bf810b930 2024-03-11T21:27:26.154 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bf57fa640 1 -- 172.21.15.129:0/1131851907 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0010760 con 0x7f5bf810b930 2024-03-11T21:27:26.154 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.167+0000 7f5bfe481640 1 -- 172.21.15.129:0/1131851907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf81a3ca0 con 0x7f5bf810b930 2024-03-11T21:27:26.155 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.168+0000 7f5bfe481640 1 -- 172.21.15.129:0/1131851907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf8068620 con 0x7f5bf810b930 2024-03-11T21:27:26.155 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.169+0000 7f5bf57fa640 1 -- 172.21.15.129:0/1131851907 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7f5bf0004020 con 0x7f5bf810b930 2024-03-11T21:27:26.155 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.169+0000 7f5bf57fa640 1 --2- 172.21.15.129:0/1131851907 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f5be006ddf0 0x7f5be00702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:26.155 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.169+0000 7f5bf7fff640 1 --2- 172.21.15.129:0/1131851907 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f5be006ddf0 0x7f5be00702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:26.155 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.169+0000 7f5bf57fa640 1 -- 172.21.15.129:0/1131851907 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(790..790 src has 254..790) v4 ==== 9407+0+0 (secure 0 0 0) 0x7f5bf008efc0 con 0x7f5bf810b930 2024-03-11T21:27:26.155 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.170+0000 7f5bf57fa640 1 -- 172.21.15.129:0/1131851907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=791}) v3 -- 0x7f5be007a000 con 0x7f5bf810b930 2024-03-11T21:27:26.155 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.170+0000 7f5bf7fff640 1 --2- 172.21.15.129:0/1131851907 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f5be006ddf0 0x7f5be00702b0 secure :-1 s=READY pgs=4059 cs=0 l=1 rev1=1 crypto rx=0x7f5be8005ab0 tx=0x7f5be8005fd0 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:26.157 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.174+0000 7f5bf57fa640 1 -- 172.21.15.129:0/1131851907 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7f5bf005f740 con 0x7f5bf810b930 2024-03-11T21:27:26.332 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.349+0000 7f5bfe481640 1 -- 172.21.15.129:0/1131851907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pool2": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5bf81a3e70 con 0x7f5bf810b930 2024-03-11T21:27:26.465 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:27:26 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: ::ffff:172.21.15.157 - - [11/Mar/2024:21:27:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:27:26.547 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.564+0000 7f5bf57fa640 1 -- 172.21.15.129:0/1131851907 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pool2": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "yes_i_really_really_mean_it": true}]=0 pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' removed v791) v1 ==== 248+0+0 (secure 0 0 0) 0x7f5bf0063d00 con 0x7f5bf810b930 2024-03-11T21:27:26.548 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.565+0000 7f5bf57fa640 1 -- 172.21.15.129:0/1131851907 <== mon.0 v2:172.21.15.129:3300/0 8 ==== osd_map(791..791 src has 254..791) v4 ==== 296+0+0 (secure 0 0 0) 0x7f5bf0059740 con 0x7f5bf810b930 2024-03-11T21:27:26.548 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.565+0000 7f5bf57fa640 1 -- 172.21.15.129:0/1131851907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=792}) v3 -- 0x7f5be007ad20 con 0x7f5bf810b930 2024-03-11T21:27:26.662 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.678+0000 7f5bfe481640 1 -- 172.21.15.129:0/1131851907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pool2": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5bf81a42c0 con 0x7f5bf810b930 2024-03-11T21:27:26.662 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.679+0000 7f5bf57fa640 1 -- 172.21.15.129:0/1131851907 <== mon.0 v2:172.21.15.129:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pool2": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "yes_i_really_really_mean_it": true}]=0 pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' does not exist v791) v1 ==== 255+0+0 (secure 0 0 0) 0x7f5bf008c710 con 0x7f5bf810b930 2024-03-11T21:27:26.662 INFO:tasks.workunit.client.0.smithi129.stderr:pool 'ff0f5cc6-20a7-4630-8c16-3bfb73edfea7' does not exist 2024-03-11T21:27:26.664 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.681+0000 7f5bfe481640 1 -- 172.21.15.129:0/1131851907 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f5be006ddf0 msgr2=0x7f5be00702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:26.664 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.681+0000 7f5bfe481640 1 --2- 172.21.15.129:0/1131851907 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f5be006ddf0 0x7f5be00702b0 secure :-1 s=READY pgs=4059 cs=0 l=1 rev1=1 crypto rx=0x7f5be8005ab0 tx=0x7f5be8005fd0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.664 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.681+0000 7f5bfe481640 1 -- 172.21.15.129:0/1131851907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf810b930 msgr2=0x7f5bf819b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:26.664 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.681+0000 7f5bfe481640 1 --2- 172.21.15.129:0/1131851907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf810b930 0x7f5bf819b9f0 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0009c90 tx=0x7f5bf000c6a0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.664 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.681+0000 7f5bfe481640 1 -- 172.21.15.129:0/1131851907 shutdown_connections 2024-03-11T21:27:26.664 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.681+0000 7f5bfe481640 1 --2- 172.21.15.129:0/1131851907 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7f5be006ddf0 0x7f5be00702b0 unknown :-1 s=CLOSED pgs=4059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.664 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.681+0000 7f5bfe481640 1 --2- 172.21.15.129:0/1131851907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bf810b930 0x7f5bf819b9f0 unknown :-1 s=CLOSED pgs=3045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.665 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.681+0000 7f5bfe481640 1 --2- 172.21.15.129:0/1131851907 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7f5bf8102e80 0x7f5bf8197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.665 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.681+0000 7f5bfe481640 1 --2- 172.21.15.129:0/1131851907 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bf8102560 0x7f5bf8197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.665 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.681+0000 7f5bfe481640 1 -- 172.21.15.129:0/1131851907 >> 172.21.15.129:0/1131851907 conn(0x7f5bf80772e0 msgr2=0x7f5bf80fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:27:26.665 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.681+0000 7f5bfe481640 1 -- 172.21.15.129:0/1131851907 shutdown_connections 2024-03-11T21:27:26.665 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.681+0000 7f5bfe481640 1 -- 172.21.15.129:0/1131851907 wait complete. 2024-03-11T21:27:26.673 INFO:tasks.workunit.client.0.smithi129.stderr:+ ceph osd pool delete 4bf093d5-8685-43fb-a522-cbae7e0b1dcf 4bf093d5-8685-43fb-a522-cbae7e0b1dcf --yes-i-really-really-mean-it 2024-03-11T21:27:26.758 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.772+0000 7fc2f5697640 1 -- 172.21.15.129:0/2270112806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2f01074f0 msgr2=0x7fc2f01078d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:26.758 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.772+0000 7fc2f5697640 1 --2- 172.21.15.129:0/2270112806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2f01074f0 0x7fc2f01078d0 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4009930 tx=0x7fc2e402bcd0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.758 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.773+0000 7fc2f5697640 1 -- 172.21.15.129:0/2270112806 shutdown_connections 2024-03-11T21:27:26.758 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.773+0000 7fc2f5697640 1 --2- 172.21.15.129:0/2270112806 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc2f0100570 0x7fc2f010e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.758 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.773+0000 7fc2f5697640 1 --2- 172.21.15.129:0/2270112806 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2f00ffbb0 0x7fc2f0100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.758 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.773+0000 7fc2f5697640 1 --2- 172.21.15.129:0/2270112806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2f01074f0 0x7fc2f01078d0 unknown :-1 s=CLOSED pgs=3046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.759 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.773+0000 7fc2f5697640 1 -- 172.21.15.129:0/2270112806 >> 172.21.15.129:0/2270112806 conn(0x7fc2f0075510 msgr2=0x7fc2f0075920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:27:26.759 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.773+0000 7fc2f5697640 1 -- 172.21.15.129:0/2270112806 shutdown_connections 2024-03-11T21:27:26.759 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.773+0000 7fc2f5697640 1 -- 172.21.15.129:0/2270112806 wait complete. 2024-03-11T21:27:26.759 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.773+0000 7fc2f5697640 1 Processor -- start 2024-03-11T21:27:26.759 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.773+0000 7fc2f5697640 1 -- start start 2024-03-11T21:27:26.759 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.773+0000 7fc2f5697640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2f00ffbb0 0x7fc2f0197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:26.759 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2f5697640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2f0100570 0x7fc2f0197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:26.759 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2f5697640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc2f01074f0 0x7fc2f019b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:26.759 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2f5697640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0110b60 con 0x7fc2f00ffbb0 2024-03-11T21:27:26.759 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2f5697640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f01109e0 con 0x7fc2f0100570 2024-03-11T21:27:26.759 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2f5697640 1 -- --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0110ce0 con 0x7fc2f01074f0 2024-03-11T21:27:26.759 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2eeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2f00ffbb0 0x7fc2f0197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:26.760 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2ef7fe640 1 --2- >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc2f01074f0 0x7fc2f019b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:26.760 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2eeffd640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2f00ffbb0 0x7fc2f0197110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52134/0 (socket says 172.21.15.129:52134) 2024-03-11T21:27:26.760 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2eeffd640 1 -- 172.21.15.129:0/4144396476 learned_addr learned my addr 172.21.15.129:0/4144396476 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-03-11T21:27:26.760 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2ee7fc640 1 --2- 172.21.15.129:0/4144396476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2f0100570 0x7fc2f0197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:26.760 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2ef7fe640 1 -- 172.21.15.129:0/4144396476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2f0100570 msgr2=0x7fc2f0197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:26.760 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2ef7fe640 1 --2- 172.21.15.129:0/4144396476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2f0100570 0x7fc2f0197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.760 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2ef7fe640 1 -- 172.21.15.129:0/4144396476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2f00ffbb0 msgr2=0x7fc2f0197110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:26.760 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2ef7fe640 1 --2- 172.21.15.129:0/4144396476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2f00ffbb0 0x7fc2f0197110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:26.760 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2ef7fe640 1 -- 172.21.15.129:0/4144396476 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f019c100 con 0x7fc2f01074f0 2024-03-11T21:27:26.760 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2eeffd640 1 --2- 172.21.15.129:0/4144396476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2f00ffbb0 0x7fc2f0197110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T21:27:26.760 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2ee7fc640 1 --2- 172.21.15.129:0/4144396476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2f0100570 0x7fc2f0197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T21:27:26.760 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2ef7fe640 1 --2- 172.21.15.129:0/4144396476 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc2f01074f0 0x7fc2f019b9d0 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800ec50 tx=0x7fc2e800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:26.760 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2d7fff640 1 -- 172.21.15.129:0/4144396476 <== mon.2 v2:172.21.15.129:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8003b70 con 0x7fc2f01074f0 2024-03-11T21:27:26.761 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2d7fff640 1 -- 172.21.15.129:0/4144396476 <== mon.2 v2:172.21.15.129:3301/0 2 ==== config(24 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fc2e8003d10 con 0x7fc2f01074f0 2024-03-11T21:27:26.761 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2f5697640 1 -- 172.21.15.129:0/4144396476 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f019c3f0 con 0x7fc2f01074f0 2024-03-11T21:27:26.761 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2d7fff640 1 -- 172.21.15.129:0/4144396476 <== mon.2 v2:172.21.15.129:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8010690 con 0x7fc2f01074f0 2024-03-11T21:27:26.761 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.774+0000 7fc2f5697640 1 -- 172.21.15.129:0/4144396476 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f01a3c80 con 0x7fc2f01074f0 2024-03-11T21:27:26.761 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.775+0000 7fc2f5697640 1 -- 172.21.15.129:0/4144396476 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f0068620 con 0x7fc2f01074f0 2024-03-11T21:27:26.762 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.776+0000 7fc2d7fff640 1 -- 172.21.15.129:0/4144396476 <== mon.2 v2:172.21.15.129:3301/0 4 ==== mgrmap(e 22) v1 ==== 91813+0+0 (secure 0 0 0) 0x7fc2e8003fc0 con 0x7fc2f01074f0 2024-03-11T21:27:26.762 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.776+0000 7fc2d7fff640 1 --2- 172.21.15.129:0/4144396476 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fc2d806ddf0 0x7fc2d80702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T21:27:26.762 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.776+0000 7fc2d7fff640 1 -- 172.21.15.129:0/4144396476 <== mon.2 v2:172.21.15.129:3301/0 5 ==== osd_map(791..791 src has 254..791) v4 ==== 9019+0+0 (secure 0 0 0) 0x7fc2e808ee90 con 0x7fc2f01074f0 2024-03-11T21:27:26.762 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.776+0000 7fc2eeffd640 1 --2- 172.21.15.129:0/4144396476 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fc2d806ddf0 0x7fc2d80702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T21:27:26.762 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.776+0000 7fc2d7fff640 1 -- 172.21.15.129:0/4144396476 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=792}) v3 -- 0x7fc2d80797e0 con 0x7fc2f01074f0 2024-03-11T21:27:26.762 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.776+0000 7fc2eeffd640 1 --2- 172.21.15.129:0/4144396476 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fc2d806ddf0 0x7fc2d80702b0 secure :-1 s=READY pgs=4060 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4000c00 tx=0x7fc2e40058d0 comp rx=0 tx=0).ready entity=mgr.24497 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T21:27:26.763 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.781+0000 7fc2d7fff640 1 -- 172.21.15.129:0/4144396476 <== mon.2 v2:172.21.15.129:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184781 (secure 0 0 0) 0x7fc2e8057030 con 0x7fc2f01074f0 2024-03-11T21:27:26.938 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:26.955+0000 7fc2f5697640 1 -- 172.21.15.129:0/4144396476 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc2f01a3e50 con 0x7fc2f01074f0 2024-03-11T21:27:26.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:26 smithi129 ceph-mon[30184]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:26.965 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:26 smithi129 ceph-mon[30184]: pgmap v1783: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:26.966 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:26 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1131851907' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pool2": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:26.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:26 smithi129 ceph-mon[42066]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:26.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:26 smithi129 ceph-mon[42066]: pgmap v1783: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:26.966 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:26 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1131851907' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pool2": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:26.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:26 smithi157 ceph-mon[33359]: from='client.14538 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:26.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:26 smithi157 ceph-mon[33359]: pgmap v1783: 188 pgs: 188 active+clean; 870 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:26.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:26 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1131851907' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pool2": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:27.566 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.583+0000 7fc2d7fff640 1 -- 172.21.15.129:0/4144396476 <== mon.2 v2:172.21.15.129:3301/0 7 ==== osd_map(792..792 src has 254..792) v4 ==== 296+0+0 (secure 0 0 0) 0x7fc2e80596e0 con 0x7fc2f01074f0 2024-03-11T21:27:27.566 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.583+0000 7fc2d7fff640 1 -- 172.21.15.129:0/4144396476 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_subscribe({osdmap=793}) v3 -- 0x7fc2d807a030 con 0x7fc2f01074f0 2024-03-11T21:27:27.572 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.589+0000 7fc2d7fff640 1 -- 172.21.15.129:0/4144396476 <== mon.2 v2:172.21.15.129:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]=0 pool '4bf093d5-8685-43fb-a522-cbae7e0b1dcf' removed v792) v1 ==== 248+0+0 (secure 0 0 0) 0x7fc2e805bd40 con 0x7fc2f01074f0 2024-03-11T21:27:27.684 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.701+0000 7fc2f5697640 1 -- 172.21.15.129:0/4144396476 --> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc2f01a42a0 con 0x7fc2f01074f0 2024-03-11T21:27:27.685 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.702+0000 7fc2d7fff640 1 -- 172.21.15.129:0/4144396476 <== mon.2 v2:172.21.15.129:3301/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]=0 pool '4bf093d5-8685-43fb-a522-cbae7e0b1dcf' does not exist v792) v1 ==== 255+0+0 (secure 0 0 0) 0x7fc2e805f790 con 0x7fc2f01074f0 2024-03-11T21:27:27.685 INFO:tasks.workunit.client.0.smithi129.stderr:pool '4bf093d5-8685-43fb-a522-cbae7e0b1dcf' does not exist 2024-03-11T21:27:27.687 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.704+0000 7fc2f5697640 1 -- 172.21.15.129:0/4144396476 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fc2d806ddf0 msgr2=0x7fc2d80702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:27.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.704+0000 7fc2f5697640 1 --2- 172.21.15.129:0/4144396476 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fc2d806ddf0 0x7fc2d80702b0 secure :-1 s=READY pgs=4060 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4000c00 tx=0x7fc2e40058d0 comp rx=0 tx=0).stop 2024-03-11T21:27:27.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.704+0000 7fc2f5697640 1 -- 172.21.15.129:0/4144396476 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc2f01074f0 msgr2=0x7fc2f019b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T21:27:27.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.704+0000 7fc2f5697640 1 --2- 172.21.15.129:0/4144396476 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc2f01074f0 0x7fc2f019b9d0 secure :-1 s=READY pgs=3109 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800ec50 tx=0x7fc2e800c6a0 comp rx=0 tx=0).stop 2024-03-11T21:27:27.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.704+0000 7fc2f5697640 1 -- 172.21.15.129:0/4144396476 shutdown_connections 2024-03-11T21:27:27.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.704+0000 7fc2f5697640 1 --2- 172.21.15.129:0/4144396476 >> [v2:172.21.15.129:6800/1191021258,v1:172.21.15.129:6801/1191021258] conn(0x7fc2d806ddf0 0x7fc2d80702b0 unknown :-1 s=CLOSED pgs=4060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:27.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.704+0000 7fc2f5697640 1 --2- 172.21.15.129:0/4144396476 >> [v2:172.21.15.129:3301/0,v1:172.21.15.129:6790/0] conn(0x7fc2f01074f0 0x7fc2f019b9d0 unknown :-1 s=CLOSED pgs=3109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:27.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.704+0000 7fc2f5697640 1 --2- 172.21.15.129:0/4144396476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2f0100570 0x7fc2f0197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:27.688 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.704+0000 7fc2f5697640 1 --2- 172.21.15.129:0/4144396476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2f00ffbb0 0x7fc2f0197110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T21:27:27.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.704+0000 7fc2f5697640 1 -- 172.21.15.129:0/4144396476 >> 172.21.15.129:0/4144396476 conn(0x7fc2f0075510 msgr2=0x7fc2f0104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T21:27:27.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.705+0000 7fc2f5697640 1 -- 172.21.15.129:0/4144396476 shutdown_connections 2024-03-11T21:27:27.689 INFO:tasks.workunit.client.0.smithi129.stderr:2024-03-11T21:27:27.705+0000 7fc2f5697640 1 -- 172.21.15.129:0/4144396476 wait complete. 2024-03-11T21:27:27.697 INFO:tasks.workunit.client.0.smithi129.stderr:+ echo OK 2024-03-11T21:27:27.698 INFO:tasks.workunit.client.0.smithi129.stdout:OK 2024-03-11T21:27:27.698 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-11T21:27:27.699 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-11T21:27:27.765 INFO:tasks.workunit:Stopping ['rados/test.sh', 'rados/test_pool_quota.sh'] on client.0... 2024-03-11T21:27:27.766 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-11T21:27:27.829 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1131851907' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pool2": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:27:27.829 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:27 smithi129 ceph-mon[30184]: osdmap e791: 8 total, 8 up, 8 in 2024-03-11T21:27:27.829 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/1131851907' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pool2": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:27.829 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:27 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4144396476' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:27.829 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:27 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:27.830 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1131851907' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pool2": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:27:27.830 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:27 smithi129 ceph-mon[42066]: osdmap e791: 8 total, 8 up, 8 in 2024-03-11T21:27:27.830 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/1131851907' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pool2": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:27.830 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:27 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4144396476' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:27.830 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:27 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:27.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1131851907' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pool2": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:27:27.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:27 smithi157 ceph-mon[33359]: osdmap e791: 8 total, 8 up, 8 in 2024-03-11T21:27:27.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/1131851907' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "pool2": "ff0f5cc6-20a7-4630-8c16-3bfb73edfea7", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:27.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:27 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4144396476' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:27.973 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:27 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:28.091 DEBUG:teuthology.parallel:result is None 2024-03-11T21:27:28.091 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-11T21:27:28.157 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-11T21:27:28.157 DEBUG:teuthology.orchestra.run.smithi129:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-11T21:27:28.231 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-11T21:27:28.231 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T21:27:28.242 INFO:tasks.cephadm:Teardown begin 2024-03-11T21:27:28.243 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:27:28.299 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:27:28.335 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T21:27:28.335 DEBUG:teuthology.orchestra.run.smithi129:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:27:28.353 DEBUG:teuthology.orchestra.run.smithi157:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:27:28.392 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T21:27:28.392 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-11T21:27:28.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mon.a 2024-03-11T21:27:28.716 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-mon[42066]: pgmap v1785: 176 pgs: 176 active+clean; 697 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:28.716 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-mon[42066]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-11T21:27:28.716 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:27:28.716 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-mon[42066]: osdmap e792: 8 total, 8 up, 8 in 2024-03-11T21:27:28.716 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-mon[42066]: from='client.? 172.21.15.129:0/4144396476' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:28.716 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-mon[42066]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:28.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:28 smithi129 systemd[1]: Stopping Ceph mon.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:27:28.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-mon[30184]: pgmap v1785: 176 pgs: 176 active+clean; 697 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:28.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-mon[30184]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-11T21:27:28.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:27:28.716 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-mon[30184]: osdmap e792: 8 total, 8 up, 8 in 2024-03-11T21:27:28.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-mon[30184]: from='client.? 172.21.15.129:0/4144396476' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:28.717 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-mon[30184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:28.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:28 smithi157 ceph-mon[33359]: pgmap v1785: 176 pgs: 176 active+clean; 697 KiB data, 872 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:28.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:28 smithi157 ceph-mon[33359]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-11T21:27:28.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:27:28.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:28 smithi157 ceph-mon[33359]: osdmap e792: 8 total, 8 up, 8 in 2024-03-11T21:27:28.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:28 smithi157 ceph-mon[33359]: from='client.? 172.21.15.129:0/4144396476' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:28.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:28 smithi157 ceph-mon[33359]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "pool2": "4bf093d5-8685-43fb-a522-cbae7e0b1dcf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:27:29.143 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-a[30161]: 2024-03-11T21:27:28.753+0000 7f2f69fc0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:27:29.143 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-a[30161]: 2024-03-11T21:27:28.753+0000 7f2f69fc0700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T21:27:29.143 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:28 smithi129 podman[160192]: 2024-03-11 21:27:28.845347163 +0000 UTC m=+0.253460989 container died 4970c2d909d20e2a6c4a3d04b81de9dea01ea397a4addf67c6c955c886605dec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-11T21:27:29.143 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:21:27:28] ENGINE Bus STOPPING 2024-03-11T21:27:29.143 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:21:27:28] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:27:29.143 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:21:27:28] ENGINE Bus STOPPED 2024-03-11T21:27:29.143 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:21:27:28] ENGINE Bus STARTING 2024-03-11T21:27:29.144 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:21:27:28] ENGINE Serving on http://:::9283 2024-03-11T21:27:29.144 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:27:28 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y[30437]: [11/Mar/2024:21:27:28] ENGINE Bus STARTED 2024-03-11T21:27:29.465 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:29 smithi129 podman[160192]: 2024-03-11 21:27:29.350806643 +0000 UTC m=+0.758920468 container cleanup 4970c2d909d20e2a6c4a3d04b81de9dea01ea397a4addf67c6c955c886605dec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-11T21:27:29.465 INFO:journalctl@ceph.mon.a.smithi129.stdout:Mar 11 21:27:29 smithi129 bash[160192]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-a 2024-03-11T21:27:29.706 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mon.a.service' 2024-03-11T21:27:29.747 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:27:29.747 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-11T21:27:29.747 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-11T21:27:29.747 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mon.c 2024-03-11T21:27:30.155 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:29 smithi129 systemd[1]: Stopping Ceph mon.c for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:27:30.155 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:30 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-c[42043]: 2024-03-11T21:27:30.090+0000 7f3a96b77700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:27:30.155 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:30 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-c[42043]: 2024-03-11T21:27:30.090+0000 7f3a96b77700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-11T21:27:30.438 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:30 smithi129 podman[160424]: 2024-03-11 21:27:30.173016461 +0000 UTC m=+0.186586689 container died 289a2a2bcc06b2926f01f6bca58af3fb23cb14d82e6bab49f32faeb2a0835d75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-c, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-11T21:27:30.715 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:30 smithi129 podman[160424]: 2024-03-11 21:27:30.563672088 +0000 UTC m=+0.577242314 container cleanup 289a2a2bcc06b2926f01f6bca58af3fb23cb14d82e6bab49f32faeb2a0835d75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-c, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T21:27:30.715 INFO:journalctl@ceph.mon.c.smithi129.stdout:Mar 11 21:27:30 smithi129 bash[160424]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-c 2024-03-11T21:27:30.967 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mon.c.service' 2024-03-11T21:27:31.004 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:27:31.004 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-11T21:27:31.004 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-11T21:27:31.005 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mon.b 2024-03-11T21:27:31.338 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:31 smithi157 systemd[1]: Stopping Ceph mon.b for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:27:31.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:31 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-b[33334]: 2024-03-11T21:27:31.354+0000 7f8e9f626700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:27:31.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:31 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-b[33334]: 2024-03-11T21:27:31.354+0000 7f8e9f626700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T21:27:31.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:31 smithi157 podman[78478]: 2024-03-11 21:27:31.477514899 +0000 UTC m=+0.258228837 container died d3ef12de3cdde8ad73645cd1cf571f8fc80aaa1764bb07bba5ad767e87c1cd86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-11T21:27:32.203 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:31 smithi157 podman[78478]: 2024-03-11 21:27:31.952502294 +0000 UTC m=+0.733216267 container cleanup d3ef12de3cdde8ad73645cd1cf571f8fc80aaa1764bb07bba5ad767e87c1cd86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-11T21:27:32.203 INFO:journalctl@ceph.mon.b.smithi157.stdout:Mar 11 21:27:31 smithi157 bash[78478]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mon-b 2024-03-11T21:27:32.355 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mon.b.service' 2024-03-11T21:27:32.407 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:27:32.407 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-11T21:27:32.407 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-11T21:27:32.407 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mgr.y 2024-03-11T21:27:32.715 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:27:32 smithi129 systemd[1]: Stopping Ceph mgr.y for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:27:33.034 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:27:32 smithi129 podman[160557]: 2024-03-11 21:27:32.768134701 +0000 UTC m=+0.149449964 container died a9b2778718397dc47690fde33a1b9ff1c56364ffcb1100eaa93365a7a628c018 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:27:33.451 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:27:33 smithi129 podman[160557]: 2024-03-11 21:27:33.14285808 +0000 UTC m=+0.524173352 container cleanup a9b2778718397dc47690fde33a1b9ff1c56364ffcb1100eaa93365a7a628c018 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-11T21:27:33.451 INFO:journalctl@ceph.mgr.y.smithi129.stdout:Mar 11 21:27:33 smithi129 bash[160557]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-y 2024-03-11T21:27:33.563 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-11T21:27:33.599 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:27:33.599 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-11T21:27:33.600 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-11T21:27:33.600 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mgr.x 2024-03-11T21:27:33.957 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 21:27:33 smithi157 systemd[1]: Stopping Ceph mgr.x for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:27:34.223 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 21:27:33 smithi157 podman[78613]: 2024-03-11 21:27:33.974371205 +0000 UTC m=+0.167435181 container died 41d4789ab85b8de7f19c53790719266841b683462f1acae8292cb1cab42ca198 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:27:34.674 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 21:27:34 smithi157 podman[78613]: 2024-03-11 21:27:34.440180092 +0000 UTC m=+0.633244108 container cleanup 41d4789ab85b8de7f19c53790719266841b683462f1acae8292cb1cab42ca198 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-11T21:27:34.674 INFO:journalctl@ceph.mgr.x.smithi157.stdout:Mar 11 21:27:34 smithi157 bash[78613]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-mgr-x 2024-03-11T21:27:34.810 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-11T21:27:34.847 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:27:34.847 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-11T21:27:34.847 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T21:27:34.848 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.0 2024-03-11T21:27:35.149 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 21:27:34 smithi129 systemd[1]: Stopping Ceph osd.0 for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:27:35.465 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 21:27:35 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-0[46443]: 2024-03-11T21:27:35.165+0000 7f002991d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:27:35.465 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 21:27:35 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-0[46443]: 2024-03-11T21:27:35.165+0000 7f002991d700 -1 osd.0 792 *** Got signal Terminated *** 2024-03-11T21:27:35.465 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 21:27:35 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-0[46443]: 2024-03-11T21:27:35.165+0000 7f002991d700 -1 osd.0 792 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:27:40.863 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 21:27:40 smithi129 podman[160690]: 2024-03-11 21:27:40.619632207 +0000 UTC m=+5.566921595 container died 7d511ca94c3701c9f9a43635b03364581973b66f1a0238d1f577dfddd5e356e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e) 2024-03-11T21:27:41.138 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 21:27:40 smithi129 podman[160690]: 2024-03-11 21:27:40.96339597 +0000 UTC m=+5.910685357 container cleanup 7d511ca94c3701c9f9a43635b03364581973b66f1a0238d1f577dfddd5e356e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-03-11T21:27:41.139 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 21:27:40 smithi129 bash[160690]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-0 2024-03-11T21:27:41.465 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 21:27:41 smithi129 podman[160712]: 2024-03-11 21:27:41.156240015 +0000 UTC m=+0.531128102 container remove 7d511ca94c3701c9f9a43635b03364581973b66f1a0238d1f577dfddd5e356e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-11T21:27:41.891 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 21:27:41 smithi129 podman[160806]: 2024-03-11 21:27:41.473575641 +0000 UTC m=+0.018673676 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:27:41.891 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 21:27:41 smithi129 podman[160806]: 2024-03-11 21:27:41.58203288 +0000 UTC m=+0.127130914 container create 4cefc86cf53858d8d74dd8b01e2bf6c1414b8661df9659c4fb34170c85bd6e34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-11T21:27:41.892 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 21:27:41 smithi129 podman[160806]: 2024-03-11 21:27:41.809501693 +0000 UTC m=+0.354599741 container init 4cefc86cf53858d8d74dd8b01e2bf6c1414b8661df9659c4fb34170c85bd6e34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux ) 2024-03-11T21:27:41.892 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 21:27:41 smithi129 podman[160806]: 2024-03-11 21:27:41.814969852 +0000 UTC m=+0.360067889 container start 4cefc86cf53858d8d74dd8b01e2bf6c1414b8661df9659c4fb34170c85bd6e34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-11T21:27:41.892 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 21:27:41 smithi129 podman[160806]: 2024-03-11 21:27:41.909377951 +0000 UTC m=+0.454475998 container attach 4cefc86cf53858d8d74dd8b01e2bf6c1414b8661df9659c4fb34170c85bd6e34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-11T21:27:42.215 INFO:journalctl@ceph.osd.0.smithi129.stdout:Mar 11 21:27:42 smithi129 podman[160806]: 2024-03-11 21:27:42.114105226 +0000 UTC m=+0.659203271 container died 4cefc86cf53858d8d74dd8b01e2bf6c1414b8661df9659c4fb34170c85bd6e34 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-11T21:27:42.688 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T21:27:42.729 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:27:42.729 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T21:27:42.729 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T21:27:42.729 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.1 2024-03-11T21:27:42.965 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 21:27:42 smithi129 systemd[1]: Stopping Ceph osd.1 for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:27:43.465 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 21:27:43 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-1[52552]: 2024-03-11T21:27:43.085+0000 7f340d02b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:27:43.465 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 21:27:43 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-1[52552]: 2024-03-11T21:27:43.085+0000 7f340d02b700 -1 osd.1 792 *** Got signal Terminated *** 2024-03-11T21:27:43.465 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 21:27:43 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-1[52552]: 2024-03-11T21:27:43.085+0000 7f340d02b700 -1 osd.1 792 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:27:48.748 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 21:27:48 smithi129 podman[160940]: 2024-03-11 21:27:48.496614582 +0000 UTC m=+5.518723792 container died 07d3d3939894bfaf53a29f7abcb1836fea9a9946026eb67ec783037f74d89c6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-11T21:27:49.144 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 21:27:48 smithi129 podman[160940]: 2024-03-11 21:27:48.865426719 +0000 UTC m=+5.887535938 container cleanup 07d3d3939894bfaf53a29f7abcb1836fea9a9946026eb67ec783037f74d89c6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-11T21:27:49.144 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 21:27:48 smithi129 bash[160940]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-1 2024-03-11T21:27:49.465 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 21:27:49 smithi129 podman[160963]: 2024-03-11 21:27:49.162116368 +0000 UTC m=+0.664040206 container remove 07d3d3939894bfaf53a29f7abcb1836fea9a9946026eb67ec783037f74d89c6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-1, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-11T21:27:49.827 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 21:27:49 smithi129 podman[161058]: 2024-03-11 21:27:49.517520705 +0000 UTC m=+0.110523217 container create fc7c863b2b25852671997d5e5cb4f68be24efa955ffec9412b54d17efc2dc486 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux ) 2024-03-11T21:27:49.827 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 21:27:49 smithi129 podman[161058]: 2024-03-11 21:27:49.425807287 +0000 UTC m=+0.018809810 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:27:49.827 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 21:27:49 smithi129 podman[161058]: 2024-03-11 21:27:49.728544675 +0000 UTC m=+0.321547179 container init fc7c863b2b25852671997d5e5cb4f68be24efa955ffec9412b54d17efc2dc486 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-11T21:27:49.827 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 21:27:49 smithi129 podman[161058]: 2024-03-11 21:27:49.734029099 +0000 UTC m=+0.327031608 container start fc7c863b2b25852671997d5e5cb4f68be24efa955ffec9412b54d17efc2dc486 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-1-deactivate, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-11T21:27:49.827 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 21:27:49 smithi129 podman[161058]: 2024-03-11 21:27:49.844954649 +0000 UTC m=+0.437957154 container attach fc7c863b2b25852671997d5e5cb4f68be24efa955ffec9412b54d17efc2dc486 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-11T21:27:50.215 INFO:journalctl@ceph.osd.1.smithi129.stdout:Mar 11 21:27:50 smithi129 podman[161058]: 2024-03-11 21:27:50.027884029 +0000 UTC m=+0.620886536 container died fc7c863b2b25852671997d5e5cb4f68be24efa955ffec9412b54d17efc2dc486 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:27:50.600 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T21:27:50.641 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:27:50.641 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T21:27:50.641 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T21:27:50.641 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.2 2024-03-11T21:27:50.965 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 21:27:50 smithi129 systemd[1]: Stopping Ceph osd.2 for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:27:51.465 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 21:27:50 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-2[58560]: 2024-03-11T21:27:50.987+0000 7f3b679ce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:27:51.465 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 21:27:50 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-2[58560]: 2024-03-11T21:27:50.987+0000 7f3b679ce700 -1 osd.2 792 *** Got signal Terminated *** 2024-03-11T21:27:51.466 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 21:27:50 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-2[58560]: 2024-03-11T21:27:50.987+0000 7f3b679ce700 -1 osd.2 792 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:27:56.782 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 21:27:56 smithi129 podman[161191]: 2024-03-11 21:27:56.504577775 +0000 UTC m=+5.627054618 container died b7aaaf6eeec13a66e6c47bf6d66e3c8a99181e325e3af0672db664c2ba4cdbec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0) 2024-03-11T21:27:56.783 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 21:27:56 smithi129 podman[161191]: 2024-03-11 21:27:56.800432855 +0000 UTC m=+5.922909705 container cleanup b7aaaf6eeec13a66e6c47bf6d66e3c8a99181e325e3af0672db664c2ba4cdbec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True) 2024-03-11T21:27:57.215 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 21:27:56 smithi129 bash[161191]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-2 2024-03-11T21:27:57.215 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 21:27:57 smithi129 podman[161216]: 2024-03-11 21:27:57.026917929 +0000 UTC m=+0.516380581 container remove b7aaaf6eeec13a66e6c47bf6d66e3c8a99181e325e3af0672db664c2ba4cdbec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-2, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-11T21:27:57.576 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 21:27:57 smithi129 podman[161313]: 2024-03-11 21:27:57.246067476 +0000 UTC m=+0.018026915 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:27:57.577 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 21:27:57 smithi129 podman[161313]: 2024-03-11 21:27:57.352829866 +0000 UTC m=+0.124789302 container create acf75ade809d4085193b218fed07f6bf6a54fb1867d4976c0e17ebcb3e2496ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-2-deactivate, RELEASE=main-2b0101e, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:27:57.577 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 21:27:57 smithi129 podman[161313]: 2024-03-11 21:27:57.588428323 +0000 UTC m=+0.360387773 container init acf75ade809d4085193b218fed07f6bf6a54fb1867d4976c0e17ebcb3e2496ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-2-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:27:57.577 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 21:27:57 smithi129 podman[161313]: 2024-03-11 21:27:57.594227378 +0000 UTC m=+0.366186822 container start acf75ade809d4085193b218fed07f6bf6a54fb1867d4976c0e17ebcb3e2496ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:27:57.860 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 21:27:57 smithi129 podman[161313]: 2024-03-11 21:27:57.705172361 +0000 UTC m=+0.477131825 container attach acf75ade809d4085193b218fed07f6bf6a54fb1867d4976c0e17ebcb3e2496ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-11T21:27:58.148 INFO:journalctl@ceph.osd.2.smithi129.stdout:Mar 11 21:27:57 smithi129 podman[161313]: 2024-03-11 21:27:57.878126403 +0000 UTC m=+0.650085849 container died acf75ade809d4085193b218fed07f6bf6a54fb1867d4976c0e17ebcb3e2496ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:27:58.426 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T21:27:58.467 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:27:58.467 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T21:27:58.467 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-11T21:27:58.468 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.3 2024-03-11T21:27:58.693 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:27:58 smithi129 systemd[1]: Stopping Ceph osd.3 for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:27:58.965 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:27:58 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3[65041]: 2024-03-11T21:27:58.813+0000 7f53a9e42700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:27:58.965 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:27:58 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3[65041]: 2024-03-11T21:27:58.813+0000 7f53a9e42700 -1 osd.3 792 *** Got signal Terminated *** 2024-03-11T21:27:58.965 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:27:58 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3[65041]: 2024-03-11T21:27:58.813+0000 7f53a9e42700 -1 osd.3 792 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:28:01.465 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:28:01 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3[65041]: 2024-03-11T21:28:01.211+0000 7f53a6436700 -1 osd.3 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:39.240753+0000 front 2024-03-11T21:27:39.240862+0000 (oldest deadline 2024-03-11T21:28:00.940891+0000) 2024-03-11T21:28:01.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:01 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:01.103+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:01.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:01 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:01.685+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:02.465 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:28:02 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3[65041]: 2024-03-11T21:28:02.184+0000 7f53a6436700 -1 osd.3 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:39.240753+0000 front 2024-03-11T21:27:39.240862+0000 (oldest deadline 2024-03-11T21:28:00.940891+0000) 2024-03-11T21:28:02.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:02 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:02.097+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:02.972 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:02 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:02.710+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:02.973 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:02 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T21:28:02.681+0000 7fa096bc4700 -1 osd.4 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.643733+0000 front 2024-03-11T21:27:37.643616+0000 (oldest deadline 2024-03-11T21:28:01.743523+0000) 2024-03-11T21:28:03.465 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:28:03 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3[65041]: 2024-03-11T21:28:03.166+0000 7f53a6436700 -1 osd.3 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:39.240753+0000 front 2024-03-11T21:27:39.240862+0000 (oldest deadline 2024-03-11T21:28:00.940891+0000) 2024-03-11T21:28:03.470 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:03.099+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:03.723 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:03.692+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:03.723 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T21:28:03.728+0000 7fa096bc4700 -1 osd.4 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.643733+0000 front 2024-03-11T21:27:37.643616+0000 (oldest deadline 2024-03-11T21:28:01.743523+0000) 2024-03-11T21:28:03.723 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:03 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:03.486+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.008363+0000 front 2024-03-11T21:27:37.008310+0000 (oldest deadline 2024-03-11T21:28:02.908245+0000) 2024-03-11T21:28:04.403 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:28:04 smithi129 podman[161444]: 2024-03-11 21:28:04.146658987 +0000 UTC m=+5.424869030 container died db5e7759c199d83efe835106e224e892933b2925b0575e4931ed685d1c5cc029 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e) 2024-03-11T21:28:04.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:04.109+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:04.715 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:28:04 smithi129 podman[161444]: 2024-03-11 21:28:04.502888342 +0000 UTC m=+5.781098380 container cleanup db5e7759c199d83efe835106e224e892933b2925b0575e4931ed685d1c5cc029 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:28:04.715 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:28:04 smithi129 bash[161444]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3 2024-03-11T21:28:04.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:04.661+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:04.973 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T21:28:04.755+0000 7fa096bc4700 -1 osd.4 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.643733+0000 front 2024-03-11T21:27:37.643616+0000 (oldest deadline 2024-03-11T21:28:01.743523+0000) 2024-03-11T21:28:04.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:04 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:04.521+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.008363+0000 front 2024-03-11T21:27:37.008310+0000 (oldest deadline 2024-03-11T21:28:02.908245+0000) 2024-03-11T21:28:05.135 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:28:04 smithi129 podman[161480]: 2024-03-11 21:28:04.821012159 +0000 UTC m=+0.671191273 container remove db5e7759c199d83efe835106e224e892933b2925b0575e4931ed685d1c5cc029 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-11T21:28:05.389 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:28:05 smithi129 podman[161579]: 2024-03-11 21:28:05.055954742 +0000 UTC m=+0.017559386 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:28:05.389 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:28:05 smithi129 podman[161579]: 2024-03-11 21:28:05.163496875 +0000 UTC m=+0.125101571 container create 376020a9c49782b0500d7250f9219cfdcfc92532165c7e7b449a53142ce5854d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-11T21:28:05.389 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:28:05 smithi129 podman[161579]: 2024-03-11 21:28:05.407206849 +0000 UTC m=+0.368811479 container init 376020a9c49782b0500d7250f9219cfdcfc92532165c7e7b449a53142ce5854d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-11T21:28:05.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:05.155+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:05.684 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:28:05 smithi129 podman[161579]: 2024-03-11 21:28:05.412958946 +0000 UTC m=+0.374563576 container start 376020a9c49782b0500d7250f9219cfdcfc92532165c7e7b449a53142ce5854d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3-deactivate, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-11T21:28:05.684 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:28:05 smithi129 podman[161579]: 2024-03-11 21:28:05.498774749 +0000 UTC m=+0.460379381 container attach 376020a9c49782b0500d7250f9219cfdcfc92532165c7e7b449a53142ce5854d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-03-11T21:28:05.965 INFO:journalctl@ceph.osd.3.smithi129.stdout:Mar 11 21:28:05 smithi129 podman[161579]: 2024-03-11 21:28:05.701956858 +0000 UTC m=+0.663561497 container died 376020a9c49782b0500d7250f9219cfdcfc92532165c7e7b449a53142ce5854d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:28:05.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:05.705+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:05.973 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T21:28:05.753+0000 7fa096bc4700 -1 osd.4 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.643733+0000 front 2024-03-11T21:27:37.643616+0000 (oldest deadline 2024-03-11T21:28:01.743523+0000) 2024-03-11T21:28:05.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:05 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:05.523+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.008363+0000 front 2024-03-11T21:27:37.008310+0000 (oldest deadline 2024-03-11T21:28:02.908245+0000) 2024-03-11T21:28:06.257 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.3.service' 2024-03-11T21:28:06.297 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:28:06.297 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-11T21:28:06.298 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-11T21:28:06.298 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.4 2024-03-11T21:28:06.473 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:06 smithi157 systemd[1]: Stopping Ceph osd.4 for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:28:06.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:06 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:06.176+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:06.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:06 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:06.546+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.008363+0000 front 2024-03-11T21:27:37.008310+0000 (oldest deadline 2024-03-11T21:28:02.908245+0000) 2024-03-11T21:28:06.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:06 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:06.674+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:06.973 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:06 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T21:28:06.676+0000 7fa09a5d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:28:06.973 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:06 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T21:28:06.676+0000 7fa09a5d0700 -1 osd.4 792 *** Got signal Terminated *** 2024-03-11T21:28:06.973 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:06 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T21:28:06.676+0000 7fa09a5d0700 -1 osd.4 792 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:28:06.973 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:06 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T21:28:06.773+0000 7fa096bc4700 -1 osd.4 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.643733+0000 front 2024-03-11T21:27:37.643616+0000 (oldest deadline 2024-03-11T21:28:01.743523+0000) 2024-03-11T21:28:07.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:07 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:07.218+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:07.973 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:07 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T21:28:07.784+0000 7fa096bc4700 -1 osd.4 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.643733+0000 front 2024-03-11T21:27:37.643616+0000 (oldest deadline 2024-03-11T21:28:01.743523+0000) 2024-03-11T21:28:07.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:07 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:07.703+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:07.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:07 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:07.537+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.008363+0000 front 2024-03-11T21:27:37.008310+0000 (oldest deadline 2024-03-11T21:28:02.908245+0000) 2024-03-11T21:28:08.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:08 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:08.216+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:08.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:08 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:08.732+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:08.973 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:08 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T21:28:08.746+0000 7fa096bc4700 -1 osd.4 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.643733+0000 front 2024-03-11T21:27:37.643616+0000 (oldest deadline 2024-03-11T21:28:01.743523+0000) 2024-03-11T21:28:08.973 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:08 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T21:28:08.746+0000 7fa096bc4700 -1 osd.4 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:43.945334+0000 front 2024-03-11T21:27:43.945175+0000 (oldest deadline 2024-03-11T21:28:08.645173+0000) 2024-03-11T21:28:08.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:08 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:08.555+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.008363+0000 front 2024-03-11T21:27:37.008310+0000 (oldest deadline 2024-03-11T21:28:02.908245+0000) 2024-03-11T21:28:09.577 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:09 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:09.247+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:09.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:09 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:09.593+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.008363+0000 front 2024-03-11T21:27:37.008310+0000 (oldest deadline 2024-03-11T21:28:02.908245+0000) 2024-03-11T21:28:09.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:09 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:09.705+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:09.973 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:09 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T21:28:09.741+0000 7fa096bc4700 -1 osd.4 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.643733+0000 front 2024-03-11T21:27:37.643616+0000 (oldest deadline 2024-03-11T21:28:01.743523+0000) 2024-03-11T21:28:09.973 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:09 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T21:28:09.741+0000 7fa096bc4700 -1 osd.4 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:43.945334+0000 front 2024-03-11T21:27:43.945175+0000 (oldest deadline 2024-03-11T21:28:08.645173+0000) 2024-03-11T21:28:10.620 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:10 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:10.287+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:10.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:10 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:10.732+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:10.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:10 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:10.636+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.008363+0000 front 2024-03-11T21:27:37.008310+0000 (oldest deadline 2024-03-11T21:28:02.908245+0000) 2024-03-11T21:28:10.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:10 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:10.636+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:46.409495+0000 front 2024-03-11T21:27:46.409378+0000 (oldest deadline 2024-03-11T21:28:10.509310+0000) 2024-03-11T21:28:10.973 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:10 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T21:28:10.771+0000 7fa096bc4700 -1 osd.4 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.643733+0000 front 2024-03-11T21:27:37.643616+0000 (oldest deadline 2024-03-11T21:28:01.743523+0000) 2024-03-11T21:28:10.973 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:10 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4[38907]: 2024-03-11T21:28:10.771+0000 7fa096bc4700 -1 osd.4 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:43.945334+0000 front 2024-03-11T21:27:43.945175+0000 (oldest deadline 2024-03-11T21:28:08.645173+0000) 2024-03-11T21:28:11.652 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:11 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:11.241+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:11.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:11 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:11.741+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:11.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:11 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:11.669+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.008363+0000 front 2024-03-11T21:27:37.008310+0000 (oldest deadline 2024-03-11T21:28:02.908245+0000) 2024-03-11T21:28:11.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:11 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:11.669+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:46.409495+0000 front 2024-03-11T21:27:46.409378+0000 (oldest deadline 2024-03-11T21:28:10.509310+0000) 2024-03-11T21:28:12.341 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:12 smithi157 podman[78747]: 2024-03-11 21:28:12.034009472 +0000 UTC m=+5.523742731 container died 4262ce8aaca3a45f66e4863b035b61dba8b0bd41b41304a9636af4baa69769fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-11T21:28:12.342 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:12 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:12.205+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:12.342 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:12 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:12.205+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:12.624 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:12 smithi157 podman[78747]: 2024-03-11 21:28:12.457708672 +0000 UTC m=+5.947441930 container cleanup 4262ce8aaca3a45f66e4863b035b61dba8b0bd41b41304a9636af4baa69769fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-2b0101e) 2024-03-11T21:28:12.624 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:12 smithi157 bash[78747]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4 2024-03-11T21:28:12.973 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:12 smithi157 podman[78773]: 2024-03-11 21:28:12.642105606 +0000 UTC m=+0.602943302 container remove 4262ce8aaca3a45f66e4863b035b61dba8b0bd41b41304a9636af4baa69769fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-2b0101e) 2024-03-11T21:28:12.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:12 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:12.700+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.008363+0000 front 2024-03-11T21:27:37.008310+0000 (oldest deadline 2024-03-11T21:28:02.908245+0000) 2024-03-11T21:28:12.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:12 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:12.700+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:46.409495+0000 front 2024-03-11T21:27:46.409378+0000 (oldest deadline 2024-03-11T21:28:10.509310+0000) 2024-03-11T21:28:12.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:12 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:12.716+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:12.974 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:12 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:12.716+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:47.649151+0000 front 2024-03-11T21:27:47.649176+0000 (oldest deadline 2024-03-11T21:28:12.349011+0000) 2024-03-11T21:28:13.312 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:13.245+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:13.312 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:13.245+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:13.312 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:13 smithi157 podman[78871]: 2024-03-11 21:28:12.979888208 +0000 UTC m=+0.018241128 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:28:13.312 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:13 smithi157 podman[78871]: 2024-03-11 21:28:13.110032658 +0000 UTC m=+0.148385586 container create 53ff33039f08322f854eb81565afb98943943bb0fbea59178f78a578e261dad1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_CLEAN=True) 2024-03-11T21:28:13.312 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:13 smithi157 podman[78871]: 2024-03-11 21:28:13.329145766 +0000 UTC m=+0.367498688 container init 53ff33039f08322f854eb81565afb98943943bb0fbea59178f78a578e261dad1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-11T21:28:13.609 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:13 smithi157 podman[78871]: 2024-03-11 21:28:13.335148841 +0000 UTC m=+0.373501767 container start 53ff33039f08322f854eb81565afb98943943bb0fbea59178f78a578e261dad1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=main-2b0101e) 2024-03-11T21:28:13.609 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:13 smithi157 podman[78871]: 2024-03-11 21:28:13.429010553 +0000 UTC m=+0.467363495 container attach 53ff33039f08322f854eb81565afb98943943bb0fbea59178f78a578e261dad1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T21:28:13.922 INFO:journalctl@ceph.osd.4.smithi157.stdout:Mar 11 21:28:13 smithi157 podman[78871]: 2024-03-11 21:28:13.627479144 +0000 UTC m=+0.665832073 container died 53ff33039f08322f854eb81565afb98943943bb0fbea59178f78a578e261dad1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-03-11T21:28:13.923 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:13.705+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:13.923 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:13.705+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:47.649151+0000 front 2024-03-11T21:27:47.649176+0000 (oldest deadline 2024-03-11T21:28:12.349011+0000) 2024-03-11T21:28:13.923 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:13.711+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.008363+0000 front 2024-03-11T21:27:37.008310+0000 (oldest deadline 2024-03-11T21:28:02.908245+0000) 2024-03-11T21:28:13.923 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:13 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:13.711+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:46.409495+0000 front 2024-03-11T21:27:46.409378+0000 (oldest deadline 2024-03-11T21:28:10.509310+0000) 2024-03-11T21:28:14.198 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.4.service' 2024-03-11T21:28:14.239 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:28:14.240 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-11T21:28:14.240 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-11T21:28:14.240 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.5 2024-03-11T21:28:14.473 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:14 smithi157 systemd[1]: Stopping Ceph osd.5 for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:28:14.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:14.286+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:14.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:14.286+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:14.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:14.723+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:14.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:14.723+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:47.649151+0000 front 2024-03-11T21:27:47.649176+0000 (oldest deadline 2024-03-11T21:28:12.349011+0000) 2024-03-11T21:28:14.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:14.571+0000 7f86eeff4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:28:14.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:14.571+0000 7f86eeff4700 -1 osd.5 792 *** Got signal Terminated *** 2024-03-11T21:28:14.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:14.571+0000 7f86eeff4700 -1 osd.5 792 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:28:14.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:14.702+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.008363+0000 front 2024-03-11T21:27:37.008310+0000 (oldest deadline 2024-03-11T21:28:02.908245+0000) 2024-03-11T21:28:14.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:14 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:14.702+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:46.409495+0000 front 2024-03-11T21:27:46.409378+0000 (oldest deadline 2024-03-11T21:28:10.509310+0000) 2024-03-11T21:28:15.700 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:15 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:15.295+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:15.700 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:15 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:15.295+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:15.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:15 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:15.717+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:15.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:15 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:15.717+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:47.649151+0000 front 2024-03-11T21:27:47.649176+0000 (oldest deadline 2024-03-11T21:28:12.349011+0000) 2024-03-11T21:28:15.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:15 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:15.739+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.008363+0000 front 2024-03-11T21:27:37.008310+0000 (oldest deadline 2024-03-11T21:28:02.908245+0000) 2024-03-11T21:28:15.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:15 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:15.739+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:46.409495+0000 front 2024-03-11T21:27:46.409378+0000 (oldest deadline 2024-03-11T21:28:10.509310+0000) 2024-03-11T21:28:16.675 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:16 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:16.345+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:16.676 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:16 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:16.345+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:16.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:16 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:16.693+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:16.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:16 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:16.693+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:47.649151+0000 front 2024-03-11T21:27:47.649176+0000 (oldest deadline 2024-03-11T21:28:12.349011+0000) 2024-03-11T21:28:16.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:16 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:16.697+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.008363+0000 front 2024-03-11T21:27:37.008310+0000 (oldest deadline 2024-03-11T21:28:02.908245+0000) 2024-03-11T21:28:16.974 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:16 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:16.697+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:46.409495+0000 front 2024-03-11T21:27:46.409378+0000 (oldest deadline 2024-03-11T21:28:10.509310+0000) 2024-03-11T21:28:17.709 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:17 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:17.379+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:17.709 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:17 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:17.379+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:17.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:17 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:17.734+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:17.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:17 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:17.734+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:47.649151+0000 front 2024-03-11T21:27:47.649176+0000 (oldest deadline 2024-03-11T21:28:12.349011+0000) 2024-03-11T21:28:17.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:17 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:17.734+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:52.349451+0000 front 2024-03-11T21:27:52.349591+0000 (oldest deadline 2024-03-11T21:28:17.049363+0000) 2024-03-11T21:28:17.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:17 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:17.726+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.008363+0000 front 2024-03-11T21:27:37.008310+0000 (oldest deadline 2024-03-11T21:28:02.908245+0000) 2024-03-11T21:28:17.974 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:17 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:17.726+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:46.409495+0000 front 2024-03-11T21:27:46.409378+0000 (oldest deadline 2024-03-11T21:28:10.509310+0000) 2024-03-11T21:28:17.974 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:17 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:17.726+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.610193+0000 front 2024-03-11T21:27:51.610247+0000 (oldest deadline 2024-03-11T21:28:16.910183+0000) 2024-03-11T21:28:18.704 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:18 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:18.390+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:18.705 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:18 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:18.390+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:18.705 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:18 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:18.390+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:18.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:18 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:18.721+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:18.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:18 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:18.721+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:47.649151+0000 front 2024-03-11T21:27:47.649176+0000 (oldest deadline 2024-03-11T21:28:12.349011+0000) 2024-03-11T21:28:18.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:18 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:18.721+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:52.349451+0000 front 2024-03-11T21:27:52.349591+0000 (oldest deadline 2024-03-11T21:28:17.049363+0000) 2024-03-11T21:28:18.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:18 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:18.723+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:37.008363+0000 front 2024-03-11T21:27:37.008310+0000 (oldest deadline 2024-03-11T21:28:02.908245+0000) 2024-03-11T21:28:18.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:18 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:18.723+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:46.409495+0000 front 2024-03-11T21:27:46.409378+0000 (oldest deadline 2024-03-11T21:28:10.509310+0000) 2024-03-11T21:28:18.973 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:18 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5[44970]: 2024-03-11T21:28:18.723+0000 7f86e75e0700 -1 osd.5 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.610193+0000 front 2024-03-11T21:27:51.610247+0000 (oldest deadline 2024-03-11T21:28:16.910183+0000) 2024-03-11T21:28:19.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:19 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:19.364+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:19.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:19 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:19.364+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:19.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:19 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:19.364+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:20.057 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:19 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:19.747+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:20.057 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:19 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:19.747+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:47.649151+0000 front 2024-03-11T21:27:47.649176+0000 (oldest deadline 2024-03-11T21:28:12.349011+0000) 2024-03-11T21:28:20.057 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:19 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:19.747+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:52.349451+0000 front 2024-03-11T21:27:52.349591+0000 (oldest deadline 2024-03-11T21:28:17.049363+0000) 2024-03-11T21:28:20.328 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:20 smithi157 podman[79003]: 2024-03-11 21:28:20.075003883 +0000 UTC m=+5.597713193 container died fd447a4edad7ce850089185595ef8a5589e4916148d0a9c93938aa0505b1a99c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-11T21:28:20.677 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:20.381+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:20.677 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:20.381+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:20.678 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:20.381+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:20.678 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:20 smithi157 podman[79003]: 2024-03-11 21:28:20.443728176 +0000 UTC m=+5.966437485 container cleanup fd447a4edad7ce850089185595ef8a5589e4916148d0a9c93938aa0505b1a99c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-11T21:28:20.678 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:20 smithi157 bash[79003]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5 2024-03-11T21:28:20.969 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:20 smithi157 podman[79025]: 2024-03-11 21:28:20.694810428 +0000 UTC m=+0.622307493 container remove fd447a4edad7ce850089185595ef8a5589e4916148d0a9c93938aa0505b1a99c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-11T21:28:20.970 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:20.788+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:20.970 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:20.788+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:47.649151+0000 front 2024-03-11T21:27:47.649176+0000 (oldest deadline 2024-03-11T21:28:12.349011+0000) 2024-03-11T21:28:20.970 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:20 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:20.788+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:52.349451+0000 front 2024-03-11T21:27:52.349591+0000 (oldest deadline 2024-03-11T21:28:17.049363+0000) 2024-03-11T21:28:21.223 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:20 smithi157 podman[79121]: 2024-03-11 21:28:20.987305353 +0000 UTC m=+0.095273848 container create 64060ad88712262275cc99a8761d09b6dca9172dd62774a9589e981b4c926c91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=main-2b0101e) 2024-03-11T21:28:21.223 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:21 smithi157 podman[79121]: 2024-03-11 21:28:20.909298299 +0000 UTC m=+0.017266806 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:28:21.223 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:21 smithi157 podman[79121]: 2024-03-11 21:28:21.20457589 +0000 UTC m=+0.312544399 container init 64060ad88712262275cc99a8761d09b6dca9172dd62774a9589e981b4c926c91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e) 2024-03-11T21:28:21.223 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:21 smithi157 podman[79121]: 2024-03-11 21:28:21.210299967 +0000 UTC m=+0.318268468 container start 64060ad88712262275cc99a8761d09b6dca9172dd62774a9589e981b4c926c91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-11T21:28:21.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:21 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:21.403+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:21.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:21 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:21.403+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:21.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:21 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:21.403+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:21.723 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:21 smithi157 podman[79121]: 2024-03-11 21:28:21.296145695 +0000 UTC m=+0.404114210 container attach 64060ad88712262275cc99a8761d09b6dca9172dd62774a9589e981b4c926c91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-11T21:28:21.724 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:21 smithi157 podman[79121]: 2024-03-11 21:28:21.49710264 +0000 UTC m=+0.605071147 container died 64060ad88712262275cc99a8761d09b6dca9172dd62774a9589e981b4c926c91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-11T21:28:22.024 INFO:journalctl@ceph.osd.5.smithi157.stdout:Mar 11 21:28:22 smithi157 podman[79167]: 2024-03-11 21:28:22.024144929 +0000 UTC m=+0.522152921 container remove 64060ad88712262275cc99a8761d09b6dca9172dd62774a9589e981b4c926c91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:28:22.025 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:21 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:21.824+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:22.025 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:21 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:21.824+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:47.649151+0000 front 2024-03-11T21:27:47.649176+0000 (oldest deadline 2024-03-11T21:28:12.349011+0000) 2024-03-11T21:28:22.025 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:21 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:21.824+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:52.349451+0000 front 2024-03-11T21:27:52.349591+0000 (oldest deadline 2024-03-11T21:28:17.049363+0000) 2024-03-11T21:28:22.029 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.5.service' 2024-03-11T21:28:22.071 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:28:22.071 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-11T21:28:22.071 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-11T21:28:22.071 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.6 2024-03-11T21:28:22.366 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:22 smithi157 systemd[1]: Stopping Ceph osd.6 for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:28:22.723 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:22 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:22.425+0000 7fe644c28700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:28:22.723 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:22 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:22.425+0000 7fe644c28700 -1 osd.6 792 *** Got signal Terminated *** 2024-03-11T21:28:22.723 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:22 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:22.425+0000 7fe644c28700 -1 osd.6 792 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:28:22.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:22 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:22.382+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:22.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:22 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:22.382+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:22.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:22 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:22.382+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:22.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:22 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:22.383+0000 7f2590cbf700 -1 osd.7 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2185 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e792) 2024-03-11T21:28:23.223 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:22 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:22.782+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:23.223 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:22 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:22.782+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:47.649151+0000 front 2024-03-11T21:27:47.649176+0000 (oldest deadline 2024-03-11T21:28:12.349011+0000) 2024-03-11T21:28:23.223 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:22 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:22.782+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:52.349451+0000 front 2024-03-11T21:27:52.349591+0000 (oldest deadline 2024-03-11T21:28:17.049363+0000) 2024-03-11T21:28:23.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:23.379+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:23.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:23.379+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:23.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:23.379+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:23.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:23.379+0000 7f2590cbf700 -1 osd.7 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2185 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e792) 2024-03-11T21:28:24.223 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:23.813+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:24.223 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:23.813+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:47.649151+0000 front 2024-03-11T21:27:47.649176+0000 (oldest deadline 2024-03-11T21:28:12.349011+0000) 2024-03-11T21:28:24.223 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:23 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:23.813+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:52.349451+0000 front 2024-03-11T21:27:52.349591+0000 (oldest deadline 2024-03-11T21:28:17.049363+0000) 2024-03-11T21:28:24.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:24.349+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:24.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:24.349+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:24.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:24.349+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:24.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:24.349+0000 7f2590cbf700 -1 osd.7 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2185 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e792) 2024-03-11T21:28:25.223 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:24.825+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:25.223 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:24.825+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:47.649151+0000 front 2024-03-11T21:27:47.649176+0000 (oldest deadline 2024-03-11T21:28:12.349011+0000) 2024-03-11T21:28:25.223 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:24 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:24.825+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:52.349451+0000 front 2024-03-11T21:27:52.349591+0000 (oldest deadline 2024-03-11T21:28:17.049363+0000) 2024-03-11T21:28:25.697 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:25.393+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:25.697 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:25.393+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:25.697 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:25.393+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:25.697 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:25.393+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6830 osd.3 since back 2024-03-11T21:28:02.927092+0000 front 2024-03-11T21:28:02.927106+0000 (oldest deadline 2024-03-11T21:28:25.227075+0000) 2024-03-11T21:28:25.697 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:25.393+0000 7f2590cbf700 -1 osd.7 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2185 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e792) 2024-03-11T21:28:25.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:25.850+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:25.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:25.850+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:47.649151+0000 front 2024-03-11T21:27:47.649176+0000 (oldest deadline 2024-03-11T21:28:12.349011+0000) 2024-03-11T21:28:25.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:25.850+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:52.349451+0000 front 2024-03-11T21:27:52.349591+0000 (oldest deadline 2024-03-11T21:28:17.049363+0000) 2024-03-11T21:28:25.973 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:25.714Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.129:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.129:8765: connect: connection refused" 2024-03-11T21:28:25.973 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:25.714Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.129:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.129:8765: connect: connection refused" 2024-03-11T21:28:25.973 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:25.717Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.129:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.129:8765: connect: connection refused" 2024-03-11T21:28:25.973 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:25.717Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.129:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.129:8765: connect: connection refused" 2024-03-11T21:28:25.974 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:25 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:25.717Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.129:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.129:8765: connect: connection refused" 2024-03-11T21:28:26.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:26 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:26.433+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:26.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:26 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:26.433+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:26.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:26 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:26.433+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:26.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:26 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:26.433+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6830 osd.3 since back 2024-03-11T21:28:02.927092+0000 front 2024-03-11T21:28:02.927106+0000 (oldest deadline 2024-03-11T21:28:25.227075+0000) 2024-03-11T21:28:26.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:26 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:26.433+0000 7f2590cbf700 -1 osd.7 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2185 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e792) 2024-03-11T21:28:27.223 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:26 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:26.806+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:36.547425+0000 front 2024-03-11T21:27:36.547486+0000 (oldest deadline 2024-03-11T21:28:01.247584+0000) 2024-03-11T21:28:27.223 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:26 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:26.806+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:47.649151+0000 front 2024-03-11T21:27:47.649176+0000 (oldest deadline 2024-03-11T21:28:12.349011+0000) 2024-03-11T21:28:27.223 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:26 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6[51058]: 2024-03-11T21:28:26.806+0000 7fe64121c700 -1 osd.6 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:52.349451+0000 front 2024-03-11T21:27:52.349591+0000 (oldest deadline 2024-03-11T21:28:17.049363+0000) 2024-03-11T21:28:27.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:27 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:27.460+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:27.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:27 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:27.460+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:27.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:27 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:27.460+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:27.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:27 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:27.460+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6830 osd.3 since back 2024-03-11T21:28:02.927092+0000 front 2024-03-11T21:28:02.927106+0000 (oldest deadline 2024-03-11T21:28:25.227075+0000) 2024-03-11T21:28:27.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:27 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:27.460+0000 7f2590cbf700 -1 osd.7 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2185 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e792) 2024-03-11T21:28:28.086 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:27 smithi157 podman[79252]: 2024-03-11 21:28:27.78914078 +0000 UTC m=+5.482247083 container died ef6f18b387fbb0e7b7bef23cf62db1cc8ec2faed15d9cda5f8a4538cb9a68125 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-11T21:28:28.410 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:28 smithi157 podman[79252]: 2024-03-11 21:28:28.201986909 +0000 UTC m=+5.895093198 container cleanup ef6f18b387fbb0e7b7bef23cf62db1cc8ec2faed15d9cda5f8a4538cb9a68125 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-2b0101e) 2024-03-11T21:28:28.410 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:28 smithi157 bash[79252]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6 2024-03-11T21:28:28.703 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:28 smithi157 podman[79274]: 2024-03-11 21:28:28.428078998 +0000 UTC m=+0.632967686 container remove ef6f18b387fbb0e7b7bef23cf62db1cc8ec2faed15d9cda5f8a4538cb9a68125 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux ) 2024-03-11T21:28:28.703 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:28 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:28.436+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:28.703 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:28 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:28.436+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:28.703 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:28 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:28.436+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:28.703 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:28 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:28.436+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6830 osd.3 since back 2024-03-11T21:28:02.927092+0000 front 2024-03-11T21:28:02.927106+0000 (oldest deadline 2024-03-11T21:28:25.227075+0000) 2024-03-11T21:28:28.704 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:28 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:28.436+0000 7f2590cbf700 -1 osd.7 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2185 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e792) 2024-03-11T21:28:28.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:28 smithi157 podman[79372]: 2024-03-11 21:28:28.720493868 +0000 UTC m=+0.114810265 container create 2b28a36061741f4b004192eb74d34ffecb30a3e5f721bf7c93213c3313583c0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-2b0101e, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-11T21:28:28.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:28 smithi157 podman[79372]: 2024-03-11 21:28:28.623377245 +0000 UTC m=+0.017693599 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:28:28.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:28 smithi157 podman[79372]: 2024-03-11 21:28:28.948343502 +0000 UTC m=+0.342659862 container init 2b28a36061741f4b004192eb74d34ffecb30a3e5f721bf7c93213c3313583c0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-11T21:28:28.973 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:28 smithi157 podman[79372]: 2024-03-11 21:28:28.954046519 +0000 UTC m=+0.348362877 container start 2b28a36061741f4b004192eb74d34ffecb30a3e5f721bf7c93213c3313583c0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-11T21:28:29.420 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:29 smithi157 podman[79372]: 2024-03-11 21:28:29.031214201 +0000 UTC m=+0.425530560 container attach 2b28a36061741f4b004192eb74d34ffecb30a3e5f721bf7c93213c3313583c0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:28:29.420 INFO:journalctl@ceph.osd.6.smithi157.stdout:Mar 11 21:28:29 smithi157 podman[79372]: 2024-03-11 21:28:29.238690484 +0000 UTC m=+0.633006843 container died 2b28a36061741f4b004192eb74d34ffecb30a3e5f721bf7c93213c3313583c0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-03-11T21:28:29.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:29 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:29.437+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:29.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:29 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:29.437+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:29.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:29 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:29.437+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:29.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:29 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:29.437+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6830 osd.3 since back 2024-03-11T21:28:02.927092+0000 front 2024-03-11T21:28:02.927106+0000 (oldest deadline 2024-03-11T21:28:25.227075+0000) 2024-03-11T21:28:29.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:29 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:29.437+0000 7f2590cbf700 -1 osd.7 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2185 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e792) 2024-03-11T21:28:29.831 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.6.service' 2024-03-11T21:28:29.872 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:28:29.872 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-11T21:28:29.873 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-11T21:28:29.873 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.7 2024-03-11T21:28:30.206 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:29 smithi157 systemd[1]: Stopping Ceph osd.7 for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:28:30.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:30 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:30.222+0000 7f25946cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:28:30.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:30 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:30.222+0000 7f25946cb700 -1 osd.7 792 *** Got signal Terminated *** 2024-03-11T21:28:30.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:30 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:30.222+0000 7f25946cb700 -1 osd.7 792 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:28:30.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:30 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:30.434+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:30.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:30 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:30.434+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:30.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:30 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:30.434+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:30.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:30 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:30.434+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6830 osd.3 since back 2024-03-11T21:28:02.927092+0000 front 2024-03-11T21:28:02.927106+0000 (oldest deadline 2024-03-11T21:28:25.227075+0000) 2024-03-11T21:28:30.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:30 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:30.434+0000 7f2590cbf700 -1 osd.7 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2185 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e792) 2024-03-11T21:28:31.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:31 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:31.450+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:31.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:31 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:31.450+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:31.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:31 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:31.450+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:31.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:31 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:31.450+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6830 osd.3 since back 2024-03-11T21:28:02.927092+0000 front 2024-03-11T21:28:02.927106+0000 (oldest deadline 2024-03-11T21:28:25.227075+0000) 2024-03-11T21:28:31.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:31 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:31.450+0000 7f2590cbf700 -1 osd.7 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2185 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e792) 2024-03-11T21:28:32.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:32 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:32.436+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:32.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:32 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:32.436+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:32.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:32 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:32.436+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:32.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:32 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:32.436+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6830 osd.3 since back 2024-03-11T21:28:02.927092+0000 front 2024-03-11T21:28:02.927106+0000 (oldest deadline 2024-03-11T21:28:25.227075+0000) 2024-03-11T21:28:32.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:32 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:32.436+0000 7f2590cbf700 -1 osd.7 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2185 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e792) 2024-03-11T21:28:33.722 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:33.470+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:33.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:33.470+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:33.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:33.470+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:33.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:33.470+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6830 osd.3 since back 2024-03-11T21:28:02.927092+0000 front 2024-03-11T21:28:02.927106+0000 (oldest deadline 2024-03-11T21:28:25.227075+0000) 2024-03-11T21:28:33.723 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:33 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:33.470+0000 7f2590cbf700 -1 osd.7 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2185 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e792) 2024-03-11T21:28:34.973 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:34.505+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6806 osd.0 since back 2024-03-11T21:27:34.724541+0000 front 2024-03-11T21:27:34.724379+0000 (oldest deadline 2024-03-11T21:28:00.624235+0000) 2024-03-11T21:28:34.973 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:34.505+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6814 osd.1 since back 2024-03-11T21:27:45.825802+0000 front 2024-03-11T21:27:45.826134+0000 (oldest deadline 2024-03-11T21:28:11.725686+0000) 2024-03-11T21:28:34.973 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:34.505+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6822 osd.2 since back 2024-03-11T21:27:51.726283+0000 front 2024-03-11T21:27:51.726262+0000 (oldest deadline 2024-03-11T21:28:17.626145+0000) 2024-03-11T21:28:34.973 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:34.505+0000 7f2590cbf700 -1 osd.7 792 heartbeat_check: no reply from 172.21.15.129:6830 osd.3 since back 2024-03-11T21:28:02.927092+0000 front 2024-03-11T21:28:02.927106+0000 (oldest deadline 2024-03-11T21:28:25.227075+0000) 2024-03-11T21:28:34.973 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:34 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7[57139]: 2024-03-11T21:28:34.505+0000 7f2590cbf700 -1 osd.7 792 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24418.0:2185 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e792) 2024-03-11T21:28:35.846 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:35 smithi157 podman[79503]: 2024-03-11 21:28:35.581086793 +0000 UTC m=+5.467495176 container died c551667e18bba4c6a882ee885d6b1d6e6308c216a0f17c3c1a557e2e0e87f629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:28:36.178 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:35 smithi157 podman[79503]: 2024-03-11 21:28:35.961654208 +0000 UTC m=+5.848062589 container cleanup c551667e18bba4c6a882ee885d6b1d6e6308c216a0f17c3c1a557e2e0e87f629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-2b0101e) 2024-03-11T21:28:36.179 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:35 smithi157 bash[79503]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7 2024-03-11T21:28:36.473 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:36 smithi157 podman[79538]: 2024-03-11 21:28:36.196139488 +0000 UTC m=+0.615943351 container remove c551667e18bba4c6a882ee885d6b1d6e6308c216a0f17c3c1a557e2e0e87f629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-2b0101e) 2024-03-11T21:28:36.798 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:36 smithi157 podman[79634]: 2024-03-11 21:28:36.428851557 +0000 UTC m=+0.018666627 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:28:36.798 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:36 smithi157 podman[79634]: 2024-03-11 21:28:36.538479782 +0000 UTC m=+0.128294853 container create c328dbe97fda4b8fcd445ea7bb69945d08b1e4acfc84807a98a02d2bbc9bccc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7-deactivate, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-11T21:28:36.798 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:36 smithi157 podman[79634]: 2024-03-11 21:28:36.724389483 +0000 UTC m=+0.314204567 container init c328dbe97fda4b8fcd445ea7bb69945d08b1e4acfc84807a98a02d2bbc9bccc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:28:36.798 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:36 smithi157 podman[79634]: 2024-03-11 21:28:36.729810046 +0000 UTC m=+0.319625116 container start c328dbe97fda4b8fcd445ea7bb69945d08b1e4acfc84807a98a02d2bbc9bccc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:28:36.798 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:36 smithi157 podman[79634]: 2024-03-11 21:28:36.815731674 +0000 UTC m=+0.405546760 container attach c328dbe97fda4b8fcd445ea7bb69945d08b1e4acfc84807a98a02d2bbc9bccc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=main-2b0101e) 2024-03-11T21:28:37.222 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:37 smithi157 podman[79634]: 2024-03-11 21:28:37.032464685 +0000 UTC m=+0.622279765 container died c328dbe97fda4b8fcd445ea7bb69945d08b1e4acfc84807a98a02d2bbc9bccc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-11T21:28:37.551 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:37 smithi157 podman[79680]: 2024-03-11 21:28:37.551997014 +0000 UTC m=+0.514568582 container remove c328dbe97fda4b8fcd445ea7bb69945d08b1e4acfc84807a98a02d2bbc9bccc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-osd-7-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e) 2024-03-11T21:28:37.551 INFO:journalctl@ceph.osd.7.smithi157.stdout:Mar 11 21:28:37 smithi157 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-11T21:28:37.554 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@osd.7.service' 2024-03-11T21:28:37.597 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:28:37.597 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-11T21:28:37.597 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-11T21:28:37.597 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@rgw.foo.a 2024-03-11T21:28:37.894 INFO:journalctl@ceph.rgw.foo.a.smithi129.stdout:Mar 11 21:28:37 smithi129 systemd[1]: Stopping Ceph rgw.foo.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:28:38.157 INFO:journalctl@ceph.rgw.foo.a.smithi129.stdout:Mar 11 21:28:37 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-rgw-foo-a[70157]: 2024-03-11T21:28:37.920+0000 7f16199ca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:28:38.158 INFO:journalctl@ceph.rgw.foo.a.smithi129.stdout:Mar 11 21:28:37 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-rgw-foo-a[70157]: 2024-03-11T21:28:37.920+0000 7f1628242ac0 -1 shutting down 2024-03-11T21:28:48.215 INFO:journalctl@ceph.rgw.foo.a.smithi129.stdout:Mar 11 21:28:47 smithi129 bash[161710]: time="2024-03-11T21:28:47Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-11T21:28:48.215 INFO:journalctl@ceph.rgw.foo.a.smithi129.stdout:Mar 11 21:28:47 smithi129 podman[161710]: 2024-03-11 21:28:47.944820993 +0000 UTC m=+10.138872472 container died 73c7fd587e6b845a91bc80dfdd39573445e430b5a1b1a4eb40cb92755fea84f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-11T21:28:48.692 INFO:journalctl@ceph.rgw.foo.a.smithi129.stdout:Mar 11 21:28:48 smithi129 podman[161710]: 2024-03-11 21:28:48.45092471 +0000 UTC m=+10.644976197 container cleanup 73c7fd587e6b845a91bc80dfdd39573445e430b5a1b1a4eb40cb92755fea84f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, ceph=True, org.label-schema.build-date=20240307) 2024-03-11T21:28:48.692 INFO:journalctl@ceph.rgw.foo.a.smithi129.stdout:Mar 11 21:28:48 smithi129 bash[161710]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-rgw-foo-a 2024-03-11T21:28:48.865 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@rgw.foo.a.service' 2024-03-11T21:28:48.904 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:28:48.904 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-11T21:28:48.904 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-11T21:28:48.904 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@prometheus.a 2024-03-11T21:28:49.197 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:48 smithi157 systemd[1]: Stopping Ceph prometheus.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:28:49.452 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:49 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:49.215Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:28:49.453 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:49 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:49.215Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T21:28:49.453 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:49 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:49.215Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T21:28:49.453 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:49 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:49.215Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T21:28:49.453 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:49 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:49.215Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T21:28:49.453 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:49 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:49.215Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T21:28:49.453 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:49 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:49.215Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T21:28:49.453 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:49 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:49.215Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T21:28:49.453 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:49 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:49.215Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T21:28:49.453 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:49 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:49.216Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T21:28:49.453 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:49 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:49.216Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T21:28:49.454 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:49 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a[67795]: ts=2024-03-11T21:28:49.216Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T21:28:49.454 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:49 smithi157 podman[79764]: 2024-03-11 21:28:49.225533495 +0000 UTC m=+0.108518013 container died 5c244fcc3377878e6859f40deb86710bb84bccd4f52fa1b843be1fc981ab632d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:28:49.454 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:49 smithi157 podman[79764]: 2024-03-11 21:28:49.46986645 +0000 UTC m=+0.352850982 container cleanup 5c244fcc3377878e6859f40deb86710bb84bccd4f52fa1b843be1fc981ab632d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:28:49.454 INFO:journalctl@ceph.prometheus.a.smithi157.stdout:Mar 11 21:28:49 smithi157 bash[79764]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T21:28:49.860 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-11T21:28:49.896 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:28:49.896 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-11T21:28:49.897 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T21:28:50.087 INFO:teuthology.orchestra.run.smithi129.stdout:Deleting cluster with fsid: 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T21:28:52.141 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 21:28:51 smithi129 systemd[1]: Stopping Ceph alertmanager.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:28:52.455 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 21:28:52 smithi129 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a[81204]: ts=2024-03-11T21:28:52.158Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:28:52.455 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 21:28:52 smithi129 podman[162235]: 2024-03-11 21:28:52.21024528 +0000 UTC m=+0.164587455 container died 92cc4ee6779d25ec680f9ab23c18d098ce7553fef5e6d82a2de7a7ee90759b8f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:28:52.715 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 21:28:52 smithi129 podman[162235]: 2024-03-11 21:28:52.572558614 +0000 UTC m=+0.526900795 container cleanup 92cc4ee6779d25ec680f9ab23c18d098ce7553fef5e6d82a2de7a7ee90759b8f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:28:52.715 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 21:28:52 smithi129 bash[162235]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T21:28:53.215 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 21:28:52 smithi129 podman[162254]: 2024-03-11 21:28:52.831927315 +0000 UTC m=+0.619975303 container remove 92cc4ee6779d25ec680f9ab23c18d098ce7553fef5e6d82a2de7a7ee90759b8f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:28:53.215 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 21:28:52 smithi129 podman[162254]: 2024-03-11 21:28:52.907683575 +0000 UTC m=+0.695731560 volume remove a106844fbf46eec90b3993bf6fca813d3c63da11b7eac7235dc39ef82356a0be 2024-03-11T21:28:53.215 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 21:28:53 smithi129 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T21:28:53.215 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 21:28:53 smithi129 systemd[1]: Stopped Ceph alertmanager.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T21:28:53.215 INFO:journalctl@ceph.alertmanager.a.smithi129.stdout:Mar 11 21:28:53 smithi129 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 2.522s CPU time. 2024-03-11T21:28:53.715 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 21:28:53 smithi129 systemd[1]: Stopping Ceph node-exporter.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:28:53.715 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 21:28:53 smithi129 podman[162369]: 2024-03-11 21:28:53.482121938 +0000 UTC m=+0.106145772 container died 92d577d3bf30dfa5034d7b23fab2a52fcb0a3a46ee06f5c7173137cbd1d6e5e0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:28:54.068 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 21:28:53 smithi129 podman[162369]: 2024-03-11 21:28:53.826653131 +0000 UTC m=+0.450677008 container cleanup 92d577d3bf30dfa5034d7b23fab2a52fcb0a3a46ee06f5c7173137cbd1d6e5e0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:28:54.068 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 21:28:53 smithi129 bash[162369]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a 2024-03-11T21:28:54.356 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 21:28:54 smithi129 podman[162388]: 2024-03-11 21:28:54.086206932 +0000 UTC m=+0.608749846 container remove 92d577d3bf30dfa5034d7b23fab2a52fcb0a3a46ee06f5c7173137cbd1d6e5e0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:28:54.356 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 21:28:54 smithi129 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:28:54.356 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 21:28:54 smithi129 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-11T21:28:54.356 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 21:28:54 smithi129 systemd[1]: Stopped Ceph node-exporter.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T21:28:54.356 INFO:journalctl@ceph.node-exporter.a.smithi129.stdout:Mar 11 21:28:54 smithi129 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 5.316s CPU time. 2024-03-11T21:28:54.824 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T21:28:55.015 INFO:teuthology.orchestra.run.smithi157.stdout:Deleting cluster with fsid: 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T21:28:56.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:28:56 smithi157 systemd[1]: Stopping Ceph iscsi.iscsi.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:28:56.972 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:28:56 smithi157 ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a[62362]: debug Shutdown received 2024-03-11T21:29:06.848 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:29:06 smithi157 bash[80189]: time="2024-03-11T21:29:06Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-11T21:29:06.848 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:29:06 smithi157 podman[80189]: 2024-03-11 21:29:06.585891399 +0000 UTC m=+10.164142525 container died c3fc2ffaecb2d30341ac4b76753804ce9110a214b9672562743fb81c39345fec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-11T21:29:07.197 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:29:06 smithi157 podman[80189]: 2024-03-11 21:29:06.9475775 +0000 UTC m=+10.525828625 container cleanup c3fc2ffaecb2d30341ac4b76753804ce9110a214b9672562743fb81c39345fec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-11T21:29:07.197 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:29:06 smithi157 bash[80189]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a 2024-03-11T21:29:07.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:29:07 smithi157 podman[80214]: 2024-03-11 21:29:07.215289317 +0000 UTC m=+0.627125852 container remove c3fc2ffaecb2d30341ac4b76753804ce9110a214b9672562743fb81c39345fec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-iscsi-iscsi-a, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T21:29:07.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:29:07 smithi157 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-11T21:29:07.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:29:07 smithi157 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-11T21:29:07.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:29:07 smithi157 systemd[1]: Stopped Ceph iscsi.iscsi.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T21:29:07.473 INFO:journalctl@ceph.iscsi.iscsi.a.smithi157.stdout:Mar 11 21:29:07 smithi157 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Consumed 4.421s CPU time. 2024-03-11T21:29:09.185 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 21:29:08 smithi157 systemd[1]: Stopping Ceph grafana.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:29:09.439 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 21:29:09 smithi157 podman[80502]: 2024-03-11 21:29:09.202397167 +0000 UTC m=+0.125688806 container died e2ff54303d9a8f0beafedddc055d08c8be8cb37836b9b98e4c759b984ca07733 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, io.buildah.version=1.32.0, io.openshift.expose-services=, version=8.5, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-11T21:29:09.696 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 21:29:09 smithi157 podman[80502]: 2024-03-11 21:29:09.530120048 +0000 UTC m=+0.453411675 container cleanup e2ff54303d9a8f0beafedddc055d08c8be8cb37836b9b98e4c759b984ca07733 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , distribution-scope=public, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8, architecture=x86_64, vendor=Red Hat, Inc., io.buildah.version=1.32.0, vcs-type=git, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, release=239.1651231664, description=Ceph Grafana Container) 2024-03-11T21:29:09.697 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 21:29:09 smithi157 bash[80502]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a 2024-03-11T21:29:09.972 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 21:29:09 smithi157 podman[80523]: 2024-03-11 21:29:09.714451125 +0000 UTC m=+0.513419313 container remove e2ff54303d9a8f0beafedddc055d08c8be8cb37836b9b98e4c759b984ca07733 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , name=ubi8, architecture=x86_64, description=Ceph Grafana Container, io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vendor=Red Hat, Inc., io.openshift.expose-services=, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-11T21:29:09.973 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 21:29:09 smithi157 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:29:09.973 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 21:29:09 smithi157 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-11T21:29:09.973 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 21:29:09 smithi157 systemd[1]: Stopped Ceph grafana.a for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T21:29:09.973 INFO:journalctl@ceph.grafana.a.smithi157.stdout:Mar 11 21:29:09 smithi157 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@grafana.a.service: Consumed 18.027s CPU time. 2024-03-11T21:29:10.387 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 21:29:10 smithi157 systemd[1]: Stopping Ceph node-exporter.b for 27bbebfa-dfe8-11ee-95c8-87774f69a715... 2024-03-11T21:29:10.666 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 21:29:10 smithi157 podman[80640]: 2024-03-11 21:29:10.405591424 +0000 UTC m=+0.123872792 container died c176bd34aee40458fbb254a777eb13107cebcedf914765a1ae427bb58a22f543 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T21:29:10.972 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 21:29:10 smithi157 podman[80640]: 2024-03-11 21:29:10.792384657 +0000 UTC m=+0.510666024 container cleanup c176bd34aee40458fbb254a777eb13107cebcedf914765a1ae427bb58a22f543 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T21:29:10.972 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 21:29:10 smithi157 bash[80640]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b 2024-03-11T21:29:11.293 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 21:29:11 smithi157 podman[80659]: 2024-03-11 21:29:11.026652686 +0000 UTC m=+0.621665349 container remove c176bd34aee40458fbb254a777eb13107cebcedf914765a1ae427bb58a22f543 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T21:29:11.293 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 21:29:11 smithi157 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:29:11.293 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 21:29:11 smithi157 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-11T21:29:11.293 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 21:29:11 smithi157 systemd[1]: Stopped Ceph node-exporter.b for 27bbebfa-dfe8-11ee-95c8-87774f69a715. 2024-03-11T21:29:11.293 INFO:journalctl@ceph.node-exporter.b.smithi157.stdout:Mar 11 21:29:11 smithi157 systemd[1]: ceph-27bbebfa-dfe8-11ee-95c8-87774f69a715@node-exporter.b.service: Consumed 5.160s CPU time. 2024-03-11T21:29:11.773 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:29:11.808 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:29:11.838 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T21:29:11.840 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591749/remote/smithi129/crash 2024-03-11T21:29:11.841 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/crash -- . 2024-03-11T21:29:11.889 INFO:teuthology.orchestra.run.smithi129.stderr:tar: /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T21:29:11.889 INFO:teuthology.orchestra.run.smithi129.stderr:tar: Error is not recoverable: exiting now 2024-03-11T21:29:11.891 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591749/remote/smithi157/crash 2024-03-11T21:29:11.892 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/crash -- . 2024-03-11T21:29:11.932 INFO:teuthology.orchestra.run.smithi157.stderr:tar: /var/lib/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T21:29:11.932 INFO:teuthology.orchestra.run.smithi157.stderr:tar: Error is not recoverable: exiting now 2024-03-11T21:29:11.933 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T21:29:11.934 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-11T21:29:11.967 INFO:teuthology.orchestra.run.smithi129.stdout:2024-03-11T20:47:20.463493+0000 mon.a (mon.0) 175 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-03-11T21:29:11.967 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T21:29:11.968 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[SEC\]' /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-11T21:29:12.040 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[ERR\]' /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-11T21:29:12.117 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[WRN\]' /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-11T21:29:12.189 INFO:teuthology.orchestra.run.smithi129.stdout:2024-03-11T20:47:20.463493+0000 mon.a (mon.0) 175 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-03-11T21:29:12.190 INFO:tasks.cephadm:Compressing logs... 2024-03-11T21:29:12.190 DEBUG:teuthology.orchestra.run.smithi129:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T21:29:12.232 DEBUG:teuthology.orchestra.run.smithi157:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T21:29:12.257 INFO:teuthology.orchestra.run.smithi129.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T21:29:12.257 INFO:teuthology.orchestra.run.smithi129.stderr:: No such file or directory 2024-03-11T21:29:12.257 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-11T21:29:12.257 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T21:29:12.257 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-11T21:29:12.257 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-mon.a.log: /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T21:29:12.259 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.3.log 2024-03-11T21:29:12.259 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-11T21:29:12.259 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T21:29:12.261 INFO:teuthology.orchestra.run.smithi157.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T21:29:12.261 INFO:teuthology.orchestra.run.smithi157.stderr:: No such file or directory 2024-03-11T21:29:12.261 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.6.log 2024-03-11T21:29:12.261 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T21:29:12.262 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.4.log 2024-03-11T21:29:12.262 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.7.log 2024-03-11T21:29:12.267 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.5.log 2024-03-11T21:29:12.267 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.4.log: /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-11T21:29:12.267 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/tcmu-runner.log 2024-03-11T21:29:12.267 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T21:29:12.267 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/tcmu-runner.log: 67.5% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/tcmu-runner.log.gz 2024-03-11T21:29:12.268 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.log 2024-03-11T21:29:12.268 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.cephadm.log: 81.6% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T21:29:12.268 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T21:29:12.268 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.log: 84.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T21:29:12.268 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-11T21:29:12.269 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-mgr.y.log: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T21:29:12.269 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T21:29:12.271 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-volume.log: 88.0% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T21:29:12.271 INFO:teuthology.orchestra.run.smithi157.stderr: 92.3%/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-mon.b.log: -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T21:29:12.274 INFO:teuthology.orchestra.run.smithi157.stderr: 92.7% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-11T21:29:12.276 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.log 2024-03-11T21:29:12.277 INFO:teuthology.orchestra.run.smithi129.stderr: 95.3%/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.cephadm.log: -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T21:29:12.277 INFO:teuthology.orchestra.run.smithi129.stderr: 89.0% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T21:29:12.279 INFO:teuthology.orchestra.run.smithi157.stderr: 91.0% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T21:29:12.297 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T21:29:12.307 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.log: 93.4%gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T21:29:12.315 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-11T21:29:12.315 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T21:29:12.327 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-mon.c.log: 94.3% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-11T21:29:12.355 INFO:teuthology.orchestra.run.smithi129.stderr: 91.0% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T21:29:12.834 INFO:teuthology.orchestra.run.smithi129.stderr: 91.0% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-11T21:29:13.861 INFO:teuthology.orchestra.run.smithi157.stderr: 92.9% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-11T21:29:13.879 INFO:teuthology.orchestra.run.smithi129.stderr: 92.9% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-11T21:29:15.713 INFO:teuthology.orchestra.run.smithi129.stderr: 92.0% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-11T21:29:19.927 INFO:teuthology.orchestra.run.smithi129.stderr: 93.8% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T21:29:20.177 INFO:teuthology.orchestra.run.smithi157.stderr: 93.9% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.5.log.gz 2024-03-11T21:29:20.334 INFO:teuthology.orchestra.run.smithi157.stderr: 93.8% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.6.log.gz 2024-03-11T21:29:20.510 INFO:teuthology.orchestra.run.smithi157.stderr: 93.9% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.4.log.gz 2024-03-11T21:29:20.658 INFO:teuthology.orchestra.run.smithi157.stderr: 93.8% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.7.log.gz 2024-03-11T21:29:20.661 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-11T21:29:20.661 INFO:teuthology.orchestra.run.smithi157.stderr:real 0m8.415s 2024-03-11T21:29:20.661 INFO:teuthology.orchestra.run.smithi157.stderr:user 0m33.038s 2024-03-11T21:29:20.661 INFO:teuthology.orchestra.run.smithi157.stderr:sys 0m1.210s 2024-03-11T21:29:21.005 INFO:teuthology.orchestra.run.smithi129.stderr: 93.9% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T21:29:21.242 INFO:teuthology.orchestra.run.smithi129.stderr: 93.9% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.3.log.gz 2024-03-11T21:29:21.694 INFO:teuthology.orchestra.run.smithi129.stderr: 93.9% -- replaced with /var/log/ceph/27bbebfa-dfe8-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T21:29:21.696 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-03-11T21:29:21.696 INFO:teuthology.orchestra.run.smithi129.stderr:real 0m9.453s 2024-03-11T21:29:21.696 INFO:teuthology.orchestra.run.smithi129.stderr:user 0m39.051s 2024-03-11T21:29:21.696 INFO:teuthology.orchestra.run.smithi129.stderr:sys 0m1.348s 2024-03-11T21:29:21.696 INFO:tasks.cephadm:Archiving logs... 2024-03-11T21:29:21.696 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591749/remote/smithi129/log 2024-03-11T21:29:21.698 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T21:29:26.960 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591749/remote/smithi157/log 2024-03-11T21:29:26.961 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T21:29:32.135 INFO:tasks.cephadm:Removing cluster... 2024-03-11T21:29:32.135 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 --force 2024-03-11T21:29:32.328 INFO:teuthology.orchestra.run.smithi129.stdout:Deleting cluster with fsid: 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T21:29:32.623 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 27bbebfa-dfe8-11ee-95c8-87774f69a715 --force 2024-03-11T21:29:32.816 INFO:teuthology.orchestra.run.smithi157.stdout:Deleting cluster with fsid: 27bbebfa-dfe8-11ee-95c8-87774f69a715 2024-03-11T21:29:33.111 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T21:29:33.111 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T21:29:33.128 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T21:29:33.143 INFO:tasks.cephadm:Teardown complete 2024-03-11T21:29:33.144 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-11T21:29:33.154 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-11T21:29:33.154 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T21:29:33.171 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T21:29:33.261 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T21:29:33.270 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T21:29:33.270 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:29:33.273 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:29:33.285 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:29:33.286 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:29:33.290 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:29:33.290 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-03-11T21:29:33.290 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 5 +4130us[+4130us] +/- 119ms 2024-03-11T21:29:33.290 INFO:teuthology.orchestra.run.smithi129.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -1872us[-1905us] +/- 54ms 2024-03-11T21:29:33.291 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 +99us[ +99us] +/- 70ms 2024-03-11T21:29:33.291 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:29:33.292 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:29:33.292 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-03-11T21:29:33.292 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 67 +3927us[+3919us] +/- 118ms 2024-03-11T21:29:33.292 INFO:teuthology.orchestra.run.smithi157.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -2009us[-2023us] +/- 54ms 2024-03-11T21:29:33.292 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 -38us[ -52us] +/- 70ms 2024-03-11T21:29:33.292 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:29:33.292 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T21:29:33.301 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T21:29:33.301 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T21:29:33.330 DEBUG:teuthology.orchestra.run.smithi129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T21:29:33.410 DEBUG:teuthology.orchestra.run.smithi157:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T21:29:33.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-11T21:29:33.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:33.516 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-11T21:29:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:33.547 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T21:29:33.557 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T21:29:33.568 INFO:teuthology.task.internal:Duration was 3108.495832 seconds 2024-03-11T21:29:33.568 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T21:29:33.603 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T21:29:33.603 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T21:29:33.605 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T21:29:33.646 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:29:33.649 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:29:34.006 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-11T21:29:34.007 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-03-11T21:29:34.007 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T21:29:34.057 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2024-03-11T21:29:34.057 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T21:29:34.082 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-11T21:29:34.082 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T21:29:34.099 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T21:29:34.152 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-11T21:29:34.152 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T21:29:34.168 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T21:29:34.593 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T21:29:34.603 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T21:29:34.603 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T21:29:34.630 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T21:29:34.657 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T21:29:34.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:29:34.674 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:29:34.695 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-03-11T21:29:34.723 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2024-03-11T21:29:34.748 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:29:34.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:34.765 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T21:29:34.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:34.803 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T21:29:34.813 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T21:29:34.814 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591749/remote/smithi129 2024-03-11T21:29:34.814 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T21:29:34.882 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591749/remote/smithi157 2024-03-11T21:29:34.882 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T21:29:34.922 INFO:teuthology.task.internal:Removing archive directory... 2024-03-11T21:29:34.922 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T21:29:34.925 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T21:29:34.971 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T21:29:34.984 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T21:29:34.984 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T21:29:34.996 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T21:29:34.996 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T21:29:34.999 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T21:29:35.011 INFO:teuthology.orchestra.run.smithi129.stdout: 265605 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 21:29 /home/ubuntu/cephtest 2024-03-11T21:29:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: 265300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 21:29 /home/ubuntu/cephtest 2024-03-11T21:29:35.027 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T21:29:35.037 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T21:29:35.071 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T21:29:35.099 DEBUG:teuthology.nuke:shortname: smithi129 2024-03-11T21:29:35.099 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:29:35.124 DEBUG:teuthology.nuke:shortname: smithi157 2024-03-11T21:29:35.124 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:29:35.128 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591749', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:26:22.448830', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLNbD0TUqosiT5YT4uhvbGH9R7CelOKrM6NIO6IpVk/ilbQrhC0ZVxwPLsr67GEliWP8EmtTKJuTmI5JDWbo2sE='} 2024-03-11T21:29:35.150 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591749', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:26:22.446954', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDvam5DfA73PCdzN8Q1OHdmrR1YiuLKklyPq47IACoU0v3cUPdkzMCYA46WGU/f/uwH+mUfOc5wtGkMY73zSqpc='} 2024-03-11T21:29:35.183 INFO:teuthology.orchestra.console.smithi129:Power off 2024-03-11T21:29:35.184 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T21:29:35.210 DEBUG:teuthology.orchestra.console.smithi129:power off output: Chassis Power Control: Down/Off 2024-03-11T21:29:35.210 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:29:35.226 INFO:teuthology.orchestra.console.smithi157:Power off 2024-03-11T21:29:35.226 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T21:29:35.236 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-11T21:29:35.248 DEBUG:teuthology.orchestra.console.smithi157:power off output: Chassis Power Control: Down/Off 2024-03-11T21:29:35.249 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:29:35.274 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2024-03-11T21:29:39.238 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:29:39.275 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:29:39.349 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is on 2024-03-11T21:29:39.385 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2024-03-11T21:29:43.354 DEBUG:teuthology.orchestra.console.smithi129:pexpect command: ipmitool -H smithi129.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:29:43.386 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T21:29:43.465 DEBUG:teuthology.orchestra.console.smithi129:check power output: Chassis Power is off 2024-03-11T21:29:43.501 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is off 2024-03-11T21:29:43.566 INFO:teuthology.orchestra.console.smithi129:Power off completed 2024-03-11T21:29:43.602 INFO:teuthology.orchestra.console.smithi157:Power off completed 2024-03-11T21:29:43.706 INFO:teuthology.run:Summary data: description: orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests} duration: 3108.4958324432373 failure_reason: '"2024-03-11T20:47:20.463493+0000 mon.a (mon.0) 175 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-11T21:29:43.706 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T21:29:43.782 INFO:teuthology.run:FAIL